Closed adterskov closed 1 year ago
@adterskov Thanks for your issue. Can you show more detail about your error? Whether you can succeed in testing the connection? We can not reproduce this issue. Maybe you can tell me your WeChat or some others and take a look together?
Can you verify if the CA is correct? For example:
Can you verify if the CA is correct? For example:
- access your gitlab from a computer without the root CA certificate installed, the browser would warn about the certificate
- install the ca , make sure the warning is gone.
It's possible to connect to our corporate GitLab only with an installed CA. I checked directly from the devlake container. After installing the CA certificate, the curl command is started to execute successfully
curl "https://gitlab.example.com/api/v4/user?access_token=gitlab-personal-access-token"
@adterskov Thanks for your issue. Can you show more detail about your error? Whether you can succeed in testing the connection? We can not reproduce this issue. Maybe you can tell me your WeChat or some others and take a look together?
As I wrote in the description, the check fails with an error
Test Failed. Please check your configuration.
There is an error in the datalake container logs
caused by: error requesting https://gitlab.example.com/api/v4/user Wraps: (2) Get "https://gitlab.example.com/api/v4/user": x509: certificate signed by unknown authority Wraps: (3) Get "https://gitlab.example.com/api/v4/user" Wraps: (4) x509: certificate signed by unknown authority
Please tell me what information is still needed?
Hi @adterskov, Would you kindly confirm if the IN_SECURE_SKIP_VERIFY
in the .env
file was set to true
please? The value of the variable was read from this file.
Hello @mindlesscloud! Yes sure
########################## Set if skip verify and connect with out trusted certificate when use https ########################## IN_SECURE_SKIP_VERIFY="true"
Hi, @adterskov , I tried to reproduce the problem on v0.16.0-beta9
, but it seemed to be working just fine.
Here is how I configured the devlake
I deleted the current installation and reinstalled version v0.16.0-beta11. GitLab connection was added successfully.
But now when I run Blueprint job, an error from GitExtractor plugin is accured.
error preparing task data for gitextractor Wraps: (2) user rejected certificate for gitlab.example.com Error types: (1) hintdetail.withDetail (2) hintdetail.withDetail (3) hintdetail.withDetail (4) git.GitError
But CA certificate is installed, and I can perform git clone command from container (I get this URL from developers tool so GitExtractor use the same one)
git clone https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git
So GitLab access token is correct and User have necessary permission. Maybe GitExtractor plugin need additional configuring?
I try to run GitExtractor localy with csv store but this option doesn't work properly
go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"
Despite the fact that the parameter output is set, it tries to connect to the database
[error] failed to initialize database, got error dial tcp: lookup mysql: no such host" panic: dial tcp: lookup mysql: no such host Wraps: (2) dial tcp Wraps: (3) lookup mysql: no such host Error types: (1) hintdetail.withDetail (2) net.OpError (3) *net.DNSError
I see, will look into it ASAP
Hi, @adterskov I tried to reproduce your problem, and it doesn't appear. Weird.
Can you check to JSON output of the pipeline and see if the gitext configuration is correct?
@klesh yes, configuration in JSON is correct. I took it for local GitExtractor running and didn't have problems with it.
go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"
I try to run GitExtractor localy with csv store but this option doesn't work properly
go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"
Despite the fact that the parameter output is set, it tries to connect to the database
[error] failed to initialize database, got error dial tcp: lookup mysql: no such host" panic: dial tcp: lookup mysql: no such host Wraps: (2) dial tcp Wraps: (3) lookup mysql: no such host Error types: (1) hintdetail.withDetail (2) net.OpError (3) *net.DNSError
Problem fixed. Dot env file with configuration was needed.
@adterskov How you resloved it, i have same issue with you, i added ca cert to container and add IN_SECURE_SKIP_VERIFY
to .env
but gitextractor still cant work
time="2023-04-26 04:25:21" level=error msg=" [pipeline service] [pipeline #10] run tasks failed
caused by: Error running task 47.
Wraps: (2) error preparing task data for gitextractor
Wraps: (3) user rejected certificate for gitlab.example.com
Wraps: (4) user rejected certificate for gitlab.example.com
Error types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *hintdetail.withDetail (4) *git.GitError
"
devlake:
image: apache/devlake:v0.16.0-beta17
ports:
- "127.0.0.1:8080:8080"
restart: always
volumes:
- ./.env:/app/.env
- ./logs:/app/logs
- ./gitlab.example.com.crt:/usr/local/share/ca-certificates/gitlab.example.com.crt
environment:
LOGGING_DIR: /app/logs
command: [ "sh", "-c", "update-ca-certificates; lake" ]
depends_on:
- mysql
Problem fixed. Dot env file with configuration was needed.
@adterskov Could you elaborate more on this? Which .env file, which values in it?
@khanh96le you may take a look at this doc:https://devlake.apache.org/docs/Troubleshooting/Configuration#failed-to-collect-data-from-the-server-with-a-self-signed-certificate
Hi @klesh, thank you for your guidance.
Following the link you provided, I couldn't resolve the issue GitExtractor task failed in a GitHub/GitLab/BitBucket blueprint
. See image below:
I am using image version devlake.docker.scarf.sh/apache/devlake:v0.18.0
, which is > version v0.15.x
suggested in the docs.
Download log files from "Historical Pipelines", the log does not show anything for debugging
time="2023-11-16 04:47:14" level=info msg=" [pipeline service] [pipeline #31] [task #263] start executing task: 263"
time="2023-11-16 04:47:14" level=info msg=" [pipeline service] [pipeline #31] [task #263] start plugin"
Notice that there're some tables in the database are successfully filled with data, such ash table "pull_requests". That means the connection to Gitlab is good. But somehow GitExtrator does not accept the certification.
@khanh96le So... what did you do extactly? did you install the rootCA and what steps did you take to verify the CA is installed correctly?
@klesh
Yes I did install a self-signed rootCA by mounting the certificate file to the container, run update-ca-certificates
, etc.
I verified it worked by checking if my rootCA exists in the /etc/ssl/certs/ca-certificate.crt
, also by running the command: curl "https://my-private-gitlab.com/api/v4/user?access_token=gitlab-personal-access-token"
inside the container.
And as I said in previous message, some information from Gitlab actually fetched successfully like the data about the pull requests.
In Stage 2, there are 2 tasks: the first one runs success, the second one runs failed.
So I believe, the problem is related to the task that used GitExtractor.
This issue may be related to this OpenSSL's issue GitExtractor uses libgit2 which uses openssl for https verification. My self-signed CA keyUsage is not set for keyCertSign, so openssl rejects it.
@khanh96le I'm not sure. How should we solve this then?
Search before asking
What happened
When I try to configure connection with self-hosted GitLab an error occurred Test Failed. Please check your configuration.
Logs in the Devlake container are following
Environment variable is set
IN_SECURE_SKIP_VERIFY=true
I've already added CA certificate in docker image and run command
/usr/sbin/update-ca-certificates
Finally I see our CA certificate on the right path
/etc/ssl/certs/ca-certificate.crt
CA certificate was installed correctly and this check completed successfully
curl "https://gitlab.example.com/api/v4/user?access_token=gitlab-personal-access-token"
I also try to solve this issue with different Devlake versions v0.16.0-beta9 v0.15.1-beta5 v0.15.0
What do you expect to happen
Connection to self-hosted GitLab will be configured correctly
How to reproduce
Reproduce the steps from the official documentation https://devlake.apache.org/docs/Configuration/GitLab
Follow the Troubleshooting guide https://devlake.apache.org/docs/Troubleshooting/Configuration#failed-to-collect-data-from-the-server-with-a-self-signed-certificate
Anything else
No response
Version
v0.15.0
Are you willing to submit PR?
Code of Conduct