[Bug][GitLab] Connection to self-hosted GitLab failed (CA certificate has already installed) #4756

Closed adterskov closed 1 year ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

When I try to configure connection with self-hosted GitLab an error occurred Test Failed. Please check your configuration.

Logs in the Devlake container are following

[GIN] 2023/03/23 - 11:47:58 | 500 | 878.823223ms | ... | POST "/plugins/gitlab/test" [GIN] 2023/03/23 - 11:48:06 | 200 | 10.393876ms | ... | GET "/blueprints" time="2023-03-23 11:48:08" level=error msg="HTTP 500 error caused by: error requesting https://gitlab.example.com/api/v4/user Wraps: (2) Get "https://gitlab.example.com/api/v4/user": x509: certificate signed by unknown authority Wraps: (3) Get "https://gitlab.example.com/api/v4/user" Wraps: (4) x509: certificate signed by unknown authority Error types: (1) hintdetail.withDetail (2) hintdetail.withDetail (3) *url.Error (4) x509.UnknownAuthorityError"

Environment variable is set IN_SECURE_SKIP_VERIFY=true

I've already added CA certificate in docker image and run command /usr/sbin/update-ca-certificates

Finally I see our CA certificate on the right path /etc/ssl/certs/ca-certificate.crt

CA certificate was installed correctly and this check completed successfully curl "https://gitlab.example.com/api/v4/user?access_token=gitlab-personal-access-token"

I also try to solve this issue with different Devlake versions v0.16.0-beta9 v0.15.1-beta5 v0.15.0

What do you expect to happen

Connection to self-hosted GitLab will be configured correctly

How to reproduce

Reproduce the steps from the official documentation https://devlake.apache.org/docs/Configuration/GitLab

Follow the Troubleshooting guide https://devlake.apache.org/docs/Troubleshooting/Configuration#failed-to-collect-data-from-the-server-with-a-self-signed-certificate

Anything else

No response

Version

v0.15.0

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@adterskov Thanks for your issue. Can you show more detail about your error? Whether you can succeed in testing the connection? We can not reproduce this issue. Maybe you can tell me your WeChat or some others and take a look together?

Can you verify if the CA is correct? For example:

access your gitlab from a computer without the root CA certificate installed, the browser would warn about the certificate
install the ca , make sure the warning is gone.

Can you verify if the CA is correct? For example:

access your gitlab from a computer without the root CA certificate installed, the browser would warn about the certificate

install the ca , make sure the warning is gone.

It's possible to connect to our corporate GitLab only with an installed CA. I checked directly from the devlake container. After installing the CA certificate, the curl command is started to execute successfully curl "https://gitlab.example.com/api/v4/user?access_token=gitlab-personal-access-token"

@adterskov Thanks for your issue. Can you show more detail about your error? Whether you can succeed in testing the connection? We can not reproduce this issue. Maybe you can tell me your WeChat or some others and take a look together?

As I wrote in the description, the check fails with an error

Test Failed. Please check your configuration.

There is an error in the datalake container logs

caused by: error requesting https://gitlab.example.com/api/v4/user Wraps: (2) Get "https://gitlab.example.com/api/v4/user": x509: certificate signed by unknown authority Wraps: (3) Get "https://gitlab.example.com/api/v4/user" Wraps: (4) x509: certificate signed by unknown authority

Please tell me what information is still needed?

Hi @adterskov, Would you kindly confirm if the IN_SECURE_SKIP_VERIFY in the .env file was set to true please? The value of the variable was read from this file.

Hello @mindlesscloud! Yes sure

########################## Set if skip verify and connect with out trusted certificate when use https ########################## IN_SECURE_SKIP_VERIFY="true"

Hi, @adterskov , I tried to reproduce the problem on v0.16.0-beta9, but it seemed to be working just fine.

Here is how I configured the devlake

I deleted the current installation and reinstalled version v0.16.0-beta11. GitLab connection was added successfully.

But now when I run Blueprint job, an error from GitExtractor plugin is accured.

error preparing task data for gitextractor Wraps: (2) user rejected certificate for gitlab.example.com Error types: (1) hintdetail.withDetail (2) hintdetail.withDetail (3) hintdetail.withDetail (4) git.GitError

But CA certificate is installed, and I can perform git clone command from container (I get this URL from developers tool so GitExtractor use the same one) git clone https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git

So GitLab access token is correct and User have necessary permission. Maybe GitExtractor plugin need additional configuring?

I try to run GitExtractor localy with csv store but this option doesn't work properly go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"

Despite the fact that the parameter output is set, it tries to connect to the database

[error] failed to initialize database, got error dial tcp: lookup mysql: no such host" panic: dial tcp: lookup mysql: no such host Wraps: (2) dial tcp Wraps: (3) lookup mysql: no such host Error types: (1) hintdetail.withDetail (2) net.OpError (3) *net.DNSError

I see, will look into it ASAP

Hi, @adterskov I tried to reproduce your problem, and it doesn't appear. Weird. Can you check to JSON output of the pipeline and see if the gitext configuration is correct?

@klesh yes, configuration in JSON is correct. I took it for local GitExtractor running and didn't have problems with it. go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"

I try to run GitExtractor localy with csv store but this option doesn't work properly go run main.go -url https://git:glpat-xxxxx@gitlab.example.com/project-1/extra/project.git -id gitlab:GitlabProject:2:2701 -output "/tmp/store"

Despite the fact that the parameter output is set, it tries to connect to the database

[error] failed to initialize database, got error dial tcp: lookup mysql: no such host" panic: dial tcp: lookup mysql: no such host Wraps: (2) dial tcp Wraps: (3) lookup mysql: no such host Error types: (1) hintdetail.withDetail (2) net.OpError (3) *net.DNSError

Problem fixed. Dot env file with configuration was needed.

@adterskov How you resloved it, i have same issue with you, i added ca cert to container and add IN_SECURE_SKIP_VERIFY to .env but gitextractor still cant work

time="2023-04-26 04:25:21" level=error msg=" [pipeline service] [pipeline #10] run tasks failed
        caused by: Error running task 47.
        Wraps: (2) error preparing task data for gitextractor
        Wraps: (3) user rejected certificate for gitlab.example.com
        Wraps: (4) user rejected certificate for gitlab.example.com
        Error types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *hintdetail.withDetail (4) *git.GitError
        "

  devlake:
    image: apache/devlake:v0.16.0-beta17
    ports:
      - "127.0.0.1:8080:8080"
    restart: always
    volumes:
      - ./.env:/app/.env
      - ./logs:/app/logs
      - ./gitlab.example.com.crt:/usr/local/share/ca-certificates/gitlab.example.com.crt
    environment:
      LOGGING_DIR: /app/logs
    command: [ "sh", "-c", "update-ca-certificates; lake" ]
    depends_on:
      - mysql

Problem fixed. Dot env file with configuration was needed.

@adterskov Could you elaborate more on this? Which .env file, which values in it?

@khanh96le you may take a look at this doc:https://devlake.apache.org/docs/Troubleshooting/Configuration#failed-to-collect-data-from-the-server-with-a-self-signed-certificate

Hi @klesh, thank you for your guidance.

Following the link you provided, I couldn't resolve the issue GitExtractor task failed in a GitHub/GitLab/BitBucket blueprint. See image below:

Screenshot 2023-11-16 at 12 48 18

I am using image version devlake.docker.scarf.sh/apache/devlake:v0.18.0, which is > version v0.15.x suggested in the docs.

Download log files from "Historical Pipelines", the log does not show anything for debugging

time="2023-11-16 04:47:14" level=info msg=" [pipeline service] [pipeline #31] [task #263] start executing task: 263"
time="2023-11-16 04:47:14" level=info msg=" [pipeline service] [pipeline #31] [task #263] start plugin"

Notice that there're some tables in the database are successfully filled with data, such ash table "pull_requests". That means the connection to Gitlab is good. But somehow GitExtrator does not accept the certification.

@khanh96le So... what did you do extactly? did you install the rootCA and what steps did you take to verify the CA is installed correctly?

@klesh Yes I did install a self-signed rootCA by mounting the certificate file to the container, run update-ca-certificates, etc. I verified it worked by checking if my rootCA exists in the /etc/ssl/certs/ca-certificate.crt, also by running the command: curl "https://my-private-gitlab.com/api/v4/user?access_token=gitlab-personal-access-token" inside the container. And as I said in previous message, some information from Gitlab actually fetched successfully like the data about the pull requests.

Screenshot 2023-11-16 at 13 19 40 In Stage 2, there are 2 tasks: the first one runs success, the second one runs failed. So I believe, the problem is related to the task that used GitExtractor.

This issue may be related to this OpenSSL's issue GitExtractor uses libgit2 which uses openssl for https verification. My self-signed CA keyUsage is not set for keyCertSign, so openssl rejects it.

@khanh96le I'm not sure. How should we solve this then?

apache / incubator-devlake