search

cirruslabs / tart

macOS and Linux VMs on Apple Silicon to use in CI and other automations
https://tart.run
Other
3.81k stars 113 forks source link

Tart Push to GitLab Registry Failing #914

Open yangyi625 opened 1 week ago

yangyi625 commented 1 week ago

image

fkorotkov commented 1 week ago

How do you authenticate with the registry?

yangyi625 commented 1 week ago

I authenticate with the registry using GitLab CI's token. The specific login command I use is:

echo "$CI_JOB_TOKEN" | tart login $CI_REGISTRY --username $CI_REGISTRY_USER --password-stdin

I discovered that commenting out the following line in the tart project's source allowed me to successfully push to the gitlab registry:

https://github.com/cirruslabs/tart/blob/c0b20932c7a04ad7517980c04d6c0f34891c2488/Sources/tart/OCI/Layerizer/DiskV2.swift#L33

This workaround resolved the issue I was experiencing. However, I haven't yet identified the specific cause of this issue.

edigaryev commented 1 week ago

I've managed to successfully push a cloned ghcr.io/cirruslabs/ubuntu@sha256:7809213cbd1847a066e7f2589862f417bec2bb277438154e78e65f3a8e6394ea VM image to GitLab Enterprise Edition 17.5.0-pre (the one that's currently runs on gitlab.com).

Could you please try pushing the same image to your GitLab server using your own credentials (and not a temporary CI ones)?

tvtamas commented 5 days ago

Hi,

I'm getting the same error as @yangyi625:

pushing config...
pushing disk... this will take a while...
21%
Error: cancelled
Attempting to re-try...
Error: cancelled
Attempting to re-try...
Error: cancelled
Attempting to re-try...
Error: AuthFailed(why: "received unexpected HTTP status code 400 while retrieving an authentication token", details: "Bad Request")

Using Tart 2.18.5, self-hosted GitLab v17.4.1 and GitLab's fork of the registry version: v4.9.0-gitlab.

The push is done by using a Personal Access Token still valid for a couple of months.

After retrying the command several tries the upload percentage does increase.

tvtamas commented 4 days ago

Downgrading to Tart version 2.16.0 fixes the push issue with GitLab.

To downgrade:

curl -O https://raw.githubusercontent.com/cirruslabs/homebrew-cli/b516c00303c4ad55f636ac3fdd689dd04ff5e46e/tart.rb
brew install --build-from-source tart.rb
yangyi625 commented 4 days ago

Hi @edigaryev ,

Using a personal token works fine, but I still prefer to use a CI token.

I tried using Mitmproxy to see which part was causing the error. The results are as follows:

all request

It looks more like it's getting a 401 error during upload. The response is as follows:

upload request

I can't quite understand why I'm encountering an authentication issue here

hblockx commented 2 days ago

try to use --concurrent 1. For us the same issue appeared and got fixed by the concurrent 1. Also I tried adding a "retry" for the blobtransmission and it also helped with pulling and pushing images. Maybe I should contribute.