GitLab CI fails whenever repo is not public #267

Open kegalo opened 1 week ago

Describe the bug

After setting up a GitLab runner, whether managed through YunoHost or not, any and all build jobs on a non-public repo fail as soon the runner attempts to pull your repo.

Context

Hardware: Hetzner dedicated root server
YunoHost version: 12.0.4.1 (but this issue existed before I switched to beta)
I have access to my server: Through SSH, through the webadmin, through IPMI
Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: Yes
- Currently on the bookworm beta, but this issue existed for me before I upgraded.
Using, or trying to install package version/branch: main

Steps to reproduce

Install GitLab
Register runner
Create repo, set it to internal or private
Add .gitlab-ci.yml

Expected behavior

The runner should pull your repo using an authentication token. The runner does attempt to use the token, but it seems like something is stopping GitLab from seeing this token - I can't find any log containing the ci-runner:token@gitlab I expected. I do find two 401s in the rails production logs matching up to the build job, though.

Logs

GitLab CI log:

remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://docs.gitlab.com/17.5/ee/topics/git/troubleshooting_git.html#error-on-git-fetch-http-basic-access-denied

/var/log/gitlab/gitlab-rails/production_json.log:

{
    "method": "GET",
    "path": "/user/repo.git/info/refs",
    "format": "*/*",
    "controller": "Repositories::GitHttpController",
    "action": "info_refs",
    "status": 401,
    "time": "2024-10-28T20:24:34.046Z",
    "params": [
        {
            "key": "service",
            "value": "git-upload-pack"
        },
        {
            "key": "repository_path",
            "value": "user/repo.git"
        }
    ],
    "correlation_id": "01JBACTFYKJWQCNDGWRZCDRR8W",
    "meta.caller_id": "Repositories::GitHttpController#info_refs",
    "meta.feature_category": "source_code_management",
    "repository_storage": "default",
    "remote_ip": "XXX",
    "ua": "gitlab-runner 17.5.2 linux/amd64",
    "request_urgency": "default",
    "target_duration_s": 1,
    "redis_calls": 2,
    "redis_duration_s": 0.000639,
    "redis_read_bytes": 398,
    "redis_write_bytes": 131,
    "redis_feature_flag_calls": 2,
    "redis_feature_flag_duration_s": 0.000639,
    "redis_feature_flag_read_bytes": 398,
    "redis_feature_flag_write_bytes": 131,
    "db_count": 3,
    "db_write_count": 0,
    "db_cached_count": 0,
    "db_txn_count": 0,
    "db_replica_txn_count": 0,
    "db_primary_txn_count": 0,
    "db_main_txn_count": 0,
    "db_ci_txn_count": 0,
    "db_main_replica_txn_count": 0,
    "db_ci_replica_txn_count": 0,
    "db_replica_count": 0,
    "db_primary_count": 3,
    "db_main_count": 3,
    "db_ci_count": 0,
    "db_main_replica_count": 0,
    "db_ci_replica_count": 0,
    "db_replica_write_count": 0,
    "db_primary_write_count": 0,
    "db_main_write_count": 0,
    "db_ci_write_count": 0,
    "db_main_replica_write_count": 0,
    "db_ci_replica_write_count": 0,
    "db_replica_cached_count": 0,
    "db_primary_cached_count": 0,
    "db_main_cached_count": 0,
    "db_ci_cached_count": 0,
    "db_main_replica_cached_count": 0,
    "db_ci_replica_cached_count": 0,
    "db_replica_wal_count": 0,
    "db_primary_wal_count": 0,
    "db_main_wal_count": 0,
    "db_ci_wal_count": 0,
    "db_main_replica_wal_count": 0,
    "db_ci_replica_wal_count": 0,
    "db_replica_wal_cached_count": 0,
    "db_primary_wal_cached_count": 0,
    "db_main_wal_cached_count": 0,
    "db_ci_wal_cached_count": 0,
    "db_main_replica_wal_cached_count": 0,
    "db_ci_replica_wal_cached_count": 0,
    "db_replica_txn_max_duration_s": 0,
    "db_primary_txn_max_duration_s": 0,
    "db_main_txn_max_duration_s": 0,
    "db_ci_txn_max_duration_s": 0,
    "db_main_replica_txn_max_duration_s": 0,
    "db_ci_replica_txn_max_duration_s": 0,
    "db_replica_txn_duration_s": 0,
    "db_primary_txn_duration_s": 0,
    "db_main_txn_duration_s": 0,
    "db_ci_txn_duration_s": 0,
    "db_main_replica_txn_duration_s": 0,
    "db_ci_replica_txn_duration_s": 0,
    "db_replica_duration_s": 0,
    "db_primary_duration_s": 0.003,
    "db_main_duration_s": 0.003,
    "db_ci_duration_s": 0,
    "db_main_replica_duration_s": 0,
    "db_ci_replica_duration_s": 0,
    "cpu_s": 0.039546,
    "mem_objects": 10424,
    "mem_bytes": 1166640,
    "mem_mallocs": 4039,
    "mem_total_bytes": 1583600,
    "pid": 2119943,
    "worker_id": "puma_5",
    "rate_limiting_gates": [],
    "db_duration_s": 0.00317,
    "view_duration_s": 0.00657,
    "duration_s": 0.02882
}
{
    "method": "GET",
    "path": "/user/repo.git/info/refs",
    "format": "*/*",
    "controller": "Repositories::GitHttpController",
    "action": "info_refs",
    "status": 401,
    "time": "2024-10-28T20:24:34.249Z",
    "params": [
        {
            "key": "service",
            "value": "git-upload-pack"
        },
        {
            "key": "repository_path",
            "value": "user/repo.git"
        }
    ],
    "correlation_id": "01JBACTG5E4DTDJFWZ0E50GMA8",
    "meta.caller_id": "Repositories::GitHttpController#info_refs",
    "meta.feature_category": "source_code_management",
    "repository_storage": "default",
    "remote_ip": "XXX",
    "ua": "gitlab-runner 17.5.2 linux/amd64",
    "request_urgency": "default",
    "target_duration_s": 1,
    "db_count": 3,
    "db_write_count": 0,
    "db_cached_count": 0,
    "db_txn_count": 0,
    "db_replica_txn_count": 0,
    "db_primary_txn_count": 0,
    "db_main_txn_count": 0,
    "db_ci_txn_count": 0,
    "db_main_replica_txn_count": 0,
    "db_ci_replica_txn_count": 0,
    "db_replica_count": 0,
    "db_primary_count": 3,
    "db_main_count": 3,
    "db_ci_count": 0,
    "db_main_replica_count": 0,
    "db_ci_replica_count": 0,
    "db_replica_write_count": 0,
    "db_primary_write_count": 0,
    "db_main_write_count": 0,
    "db_ci_write_count": 0,
    "db_main_replica_write_count": 0,
    "db_ci_replica_write_count": 0,
    "db_replica_cached_count": 0,
    "db_primary_cached_count": 0,
    "db_main_cached_count": 0,
    "db_ci_cached_count": 0,
    "db_main_replica_cached_count": 0,
    "db_ci_replica_cached_count": 0,
    "db_replica_wal_count": 0,
    "db_primary_wal_count": 0,
    "db_main_wal_count": 0,
    "db_ci_wal_count": 0,
    "db_main_replica_wal_count": 0,
    "db_ci_replica_wal_count": 0,
    "db_replica_wal_cached_count": 0,
    "db_primary_wal_cached_count": 0,
    "db_main_wal_cached_count": 0,
    "db_ci_wal_cached_count": 0,
    "db_main_replica_wal_cached_count": 0,
    "db_ci_replica_wal_cached_count": 0,
    "db_replica_txn_max_duration_s": 0,
    "db_primary_txn_max_duration_s": 0,
    "db_main_txn_max_duration_s": 0,
    "db_ci_txn_max_duration_s": 0,
    "db_main_replica_txn_max_duration_s": 0,
    "db_ci_replica_txn_max_duration_s": 0,
    "db_replica_txn_duration_s": 0,
    "db_primary_txn_duration_s": 0,
    "db_main_txn_duration_s": 0,
    "db_ci_txn_duration_s": 0,
    "db_main_replica_txn_duration_s": 0,
    "db_ci_replica_txn_duration_s": 0,
    "db_replica_duration_s": 0,
    "db_primary_duration_s": 0.002,
    "db_main_duration_s": 0.002,
    "db_ci_duration_s": 0,
    "db_main_replica_duration_s": 0,
    "db_ci_replica_duration_s": 0,
    "cpu_s": 0.024558,
    "mem_objects": 9414,
    "mem_bytes": 1156880,
    "mem_mallocs": 3905,
    "mem_total_bytes": 1533440,
    "pid": 2119945,
    "worker_id": "puma_6",
    "rate_limiting_gates": [],
    "db_duration_s": 0.00197,
    "view_duration_s": 0.00496,
    "duration_s": 0.0167
}

Does it work on a public repository?

Can you provide the cli used to register the runner? How did you get your token?

It's weird because it works at home™.

It does work on a public repo, and it does work when GitLab is not installed alongside YunoHost.

I used the interactive registration and am using the docker executor.

Right now, I'm using the docker image GitLab provides (gitlab/gitlab-runner), but the issue also existed for me when I used the GitLab runner provided by the app catalog.

I got the token by going through Admin > CI/CD > Runners > New instance Runner.

I reinstalled GitLab a number of times, and tried modifying various nginx configs just to see if anything would change, but nothing did change. I reset everything back, and it still doesn't work.

Let me know if there are any more logs I can provide - I am honestly at a loss at this point, none of the logs I went through gave me any idea of why it isn't working for me.

Ok, I spoke too fast, I can reproduce your issue on my server. It'll be way easier for me to debug.

It's probably because of the Yunohost's sso.

I manually fix this issue by hacking the ssowatt conf file and adding to gitlab.main permission : "protect_against_basic_auth_spoofing": false

If you don't know what I'm talking about, just wait the next release ;)

works perfectly for me, thank you!

FYI, the conf will be overwritten on each app install/upgrade/remove

YunoHost-Apps / gitlab_ynh