Gitlab Private Repo

[mkauzlar]

Is there any documentation on how to pull from private Gitlab repos?

we have set: export HUSKYCI_CLIENT_REPO_URL="git@gitlab.com:path/to/git/repo.git" export HUSKYCI_API_GIT_PRIVATE_SSH_KEY="/home/huskyci/.ssh/id_rsa"

however we always get: [HUSKYCI][ERROR] Monitoring analysis ZsEW5l2w5k8weJXronrW8qKJ8DOApW8R: huskyCI encountered an error trying to execute this analysis: error cloning

Originally posted by @mkauzlar in https://github.com/globocom/huskyCI/issues/472#issuecomment-666250730

[Krlier]

Hi, @mkauzlar! Thanks for reaching out to us!

We do have some documentation on pulling from private repos, you can find it here: https://huskyci.opensource.globo.com/docs/guides/accessing-internal-repositories

Just as a quick tip by having a look at how you set the HUSKYCI_CLIENT_REPO_URL variable, I believe that for GitLab it should be: export HUSKYCI_CLIENT_REPO_URL=gitlab@gitlab.example.com

[mkauzlar]

hi @Krlier

we have tried every combination of the env variables without success. It looks as if the client doesn't switch or use ssh but just tries https.

Gitlab gives us the following URLs for cloning (we use git as user):

HTTPS https://gitlab.com/company/group/huskyci-test.git

SSH git@gitlab.com:company/group/huskyci-test.git

cloning on command line using git and ssh works without problems using the SSH URL.

Could you please describe here how would you set the following env variables based on the above URLs?

export HUSKYCI_CLIENT_REPO_URL="" export HUSKYCI_API_GIT_SSH_URL="" export HUSKYCI_API_GIT_URL_TO_SUBSTITUTE="" export HUSKYCI_API_GIT_PRIVATE_SSH_KEY="/path/to/private/key"

the above variables are set on our client side where the huskyci-client-bin command is run.

Is there any env variable that has to be set on the server side?

Thank you

[Krlier]

Hey, @mkauzlar!

I would set the variables like this:

export HUSKYCI_CLIENT_REPO_URL="https://gitlab.com/company/group/huskyci-test.git"
export HUSKYCI_API_GIT_SSH_URL="gitlab@gitlab.com"
export HUSKYCI_API_GIT_URL_TO_SUBSTITUTE="https://gitlab.example.com/"
export HUSKYCI_API_GIT_PRIVATE_SSH_KEY="/path/to/private/key"

It's important to notice that both HUSKYCI_API_GIT_SSH_URL and HUSKYCI_API_GIT_URL_TO_SUBSTITUTE are currently only working for the GoSec security test. If the project you're working on does not have GoLang and keeps outputting this error when trying to run huskyCI, we'd be happy to review any PRs you send our way to address that issue! 🙂

[mkauzlar]

So we don't need HUSKYCI_API_GIT_SSH_URL and HUSKYCI_API_GIT_URL_TO_SUBSTITUTE since the project doesn't use GoLang.

This leaves the config as follows:

export HUSKYCI_CLIENT_REPO_URL="https://gitlab.com/company/group/huskyci-test.git"
export HUSKYCI_API_GIT_PRIVATE_SSH_KEY="/path/to/private/key"

which doesn't work, we still get the "error cloning" message:

2020/08/05 05:54:35 {"version":"1.1","host":"42c7074c3fef","short_message":"Error clonning the following repository and branch: [https://gitlab.com/company/group/huskyci-test.git master error cloning]","full_message":"Error clonning the following repository and branch: [https://gitlab.com/company/group/huskyci-test.git master error cloning]","timestamp":1596606875,"level":3,"action":"analyze","app":"undefined","file":"/go/src/github.com/globocom/huskyCI/api/log/log.go","info":"SECURITYTEST","line":51,"tags":"undefined"}

Regarding your suggestion of how to HUSKYCI_API_GIT_SSH_URL and HUSKYCI_API_GIT_URL_TO_SUBSTITUTE I'm not sure why you wrote the values as "gitlab@gitlab.com" and "https://gitlab.example.com/" respectively.

[rafaveira3]

Hello, @mkauzlar! Glad to have you here! 🙃 Let me try to help you as well.

There are env vars that are used in client-side (HUSKYCI_CLIENT_*) and others that need to be set in the API (HUSKYCI_API_*) itself. Let me share with you a template configuration to gitlab-ci.yml that works for us here (client-side):

stages:
  - tests

huskyCI: 
  stage: tests
  image: golang:1.13-alpine
  before_script:
    - ""
  variables:
    HUSKYCI_CLIENT_URL: https://huskyci-client.myinternalcompanyurl.com
    HUSKYCI_CLIENT_API_ADDR: https://huskyci.myinternalcompanyurl.com
    HUSKYCI_CLIENT_REPO_URL: gitlab@gitlab.myinternalcompanyurl.com:$CI_PROJECT_PATH.git
    HUSKYCI_CLIENT_REPO_BRANCH: $CI_COMMIT_REF_NAME
    HUSKYCI_CLIENT_API_USE_HTTPS: "true"
  script:
    - wget $HUSKYCI_CLIENT_URL/huskyci-client
    - chmod +x huskyci-client
    - ./huskyci-client
  allow_failure: true

The environment HUSKYCI_API_GIT_PRIVATE_SSH_KEY needs to be set in the server-side (Golang API) and it is not the path where the ssh key is, but the content itself of the SSH key:

user@huskyserver:/home/application/current$ echo $HUSKYCI_API_GIT_PRIVATE_SSH_KEY
-----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5v ........ (REMOVED)

If you are using tsuru to deploy huskyCI API you can set this environment variable using the following command:

tsuru env-set -a huskyci --private HUSKYCI_API_GIT_PRIVATE_SSH_KEY="$(cat /path/to/sshkey)"

Otherwise:

export HUSKYCI_API_GIT_PRIVATE_SSH_KEY="$(cat /path/to/sshkey)"

Please, reply to this issue if you are still having some issues cloning internal repositories in your huskyCI installation. 🙃

[mkauzlar]

hi @rafaveira3 and thanks for jumping in. Finally I got the answer from my previous post, that HUSKYCI_API_GIT_PRIVATE_SSH_KEY is to be set on the server and not on the client (something that should be the other way around IMHO)

We don't use tsuru. We are deploying huskyCI on a local machine as development environment as per this documentation: https://huskyci.opensource.globo.com/docs/development/set-up-environment#installing

We wanted to test this local deployment against one of our code repositories on Gitlab.

Based on your suggestion we edited the docker-compose.yml and added HUSKYCI_API_GIT_PRIVATE_SSH_KEY: "private key content" After this we did the: make install To verify that the variable was present in the huskyCI_API container we did login into it: docker exec -it huskyCI_API /bin/bash and indeed we saw:

# env
.....
HUSKYCI_API_GIT_PRIVATE_SSH_KEY=-----BEGIN RSA PRIVATE KEY-----MIIEowIBA.....
.....

Anyhow the error cloning remains

Is there any command we can run in the huskyCI_API container in order to test and debug this?

[rafaveira3]

Hey, @mkauzlar! I have finally got it!

We should be fine after we merge #502! Feel free to test now your internal repositories using this branch or wait until we got it into master. Please check the Testing section on how I managed to get this one working by here!

Thanks a lot for reporting this issue to us! 🙃

[Krlier]

Thanks for the contribution, @rafaveira3!

@mkauzlar, would you mind giving it another try with the changes recently added in #502? If you're still facing some difficulties running huskyCI, please let us know and we'll reopen this issue. 🙂