search

goharbor / acceleration-service

Provides a general service to support image acceleration based on kinds of accelerator like Nydus and eStargz etc.
Apache License 2.0
73 stars 26 forks source link

Can i convert proxyfied registries ? #249

Open ptempier opened 8 months ago

ptempier commented 8 months ago

I mean, i tried with estargz, it didn't work, but i was somehow expecting it to work. Not sure if it's not supported at all and i am doing something wrong. Maybe i need a different setup, like an actual proxy project and an accelerated copy of that project. It's not working and its not supposed to work. It doesn't work right now, but maybe it will in a future release.

Desiki-high commented 8 months ago

Hi @ptempier. Could you please give more information to help reproduce your issue? Such as your setup steps and what kind of proxyfied registries are used. Acceleration-Service can work well with estargz in the latest harbor.

ptempier commented 8 months ago

Hello

I am not using the latest version, but v2.9.0-6d1ad65c, so maybe that's it. I tried looking at the changlog but couldn't find anything related to acceld and proxy. Was the support added in 2.10 ?

harbor-acceld is v0.2.13

The test was doing a pull from a proxified github registry. The robot could successfully authenticate to pull the image the accelertor could transform to estargz but i was getting an nginx permission error (403?) when pushing, it was not appearing in the harbor logs.

Creating an estargz image via kaniko, pushing it in a different projet then pulling it with a properly configured containerd works as expected.

Desiki-high commented 8 months ago

Can you provide the acceld work log? @ptempier

Desiki-high commented 8 months ago

Actually harbor v2.9 is supported too.

Desiki-high commented 8 months ago

Can you provide the acceld work log? @ptempier

If you can provide the harbor log, it will be better.

ptempier commented 8 months ago

The time it almost worked , i had this error :

error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"

Just tried a docker pull anonyme/hub.docker.com/library/postgres It went through the proxy cache as per habor log

harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:3a27b8f06bc0cc0b76ab124b8c48bf3177703aedbd9cc28fcebc0e312bcb8c7a artifact create 1/9/24, 12:51 PM
anonymous hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact pull 1/9/24, 12:45 PM
harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact create 1/9/24, 12:45 PM

It went through the webhook

10703 WEBHOOK Success Artifact pushed xxxx 1/9/24, 12:51 PM 1/9/24, 12:51 PM
10702 WEBHOOK Success Artifact pushed xxx 1/9/24, 12:45 PM 1/9/24, 12:45 PM

But then in acceld failed with : 

time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"
time="2024-01-08T15:40:46.676196062Z" level=info msg="Version: v0.2.13 b82ccee1e73845741b033f8b04ab418fad5b84ef.20240108.0649\n"
time="2024-01-08T15:40:46.698308247Z" level=info msg="[API] HTTP server started on 0.0.0.0:2077"
time="2024-01-08T15:50:26.552634578Z" level=info msg="received webhook request from 172.19.0.1:52386" module=api
time="2024-01-08T15:50:26.581593392Z" level=info msg="POST /api/v1/conversions 200 28.919643ms 602>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:26.606109487Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/rockylinux@sha256:8b5296204bad12e84837c7b4c8b2cdff45bcc92ee5ffaaa3e86ba683f2384b14"
time="2024-01-08T15:50:31.651439807Z" level=info msg="received webhook request from 172.19.0.1:52392" module=api
time="2024-01-08T15:50:31.659627035Z" level=info msg="POST /api/v1/conversions 200 8.161151ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:31.668051229Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:28a16e31b140d750048cd5fadcaed22ac08d0eeb18567f79f822aee1f237b43c"
time="2024-01-08T15:53:45.357608101Z" level=info msg="received webhook request from 172.19.0.1:57896" module=api
time="2024-01-08T15:53:45.366814898Z" level=info msg="POST /api/v1/conversions 200 9.191926ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:53:45.373215343Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/nginx@sha256:5be1749f6a023b14ef069f2bbe1afd9a39295694a490963e527a848e4bc4d442"
time="2024-01-08T15:56:51.580870348Z" level=info msg="received webhook request from 172.19.0.1:47690" module=api
time="2024-01-08T15:56:51.587785057Z" level=info msg="POST /api/v1/conversions 200 6.842051ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:56:51.634318232Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:6f453b1c7bcbc42b8e3d7949d8dfa28a70f8bf86dff277f5909c5e714ee5153a"
time="2024-01-08T16:05:49.7946499Z" level=info msg="received webhook request from 127.0.0.1:34366" module=api
time="2024-01-08T16:05:49.80163495Z" level=info msg="POST /api/v1/conversions?sync=false 200 7.002226ms 135>5bytes 127.0.0.1" module=api
time="2024-01-08T16:05:49.802579063Z" level=info msg="pulling image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:05:52.887395575Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:05:52.888131555Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:05:53.009889983Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:05:53.009965038Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:06:05.439993598Z" level=info msg="pulled image anonyme/hub.docker.com/library/mysql:latest , elapse 15.607270101s" module=converter
time="2024-01-08T16:06:05.440048067Z" level=info msg="converting image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:06:13.090493323Z" level=info msg="received webhook request from 172.19.0.1:51998" module=api
time="2024-01-08T16:06:13.09860536Z" level=info msg="received webhook request from 172.19.0.1:51990" module=api
time="2024-01-08T16:06:13.11688972Z" level=info msg="POST /api/v1/conversions 200 26.377558ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.166796733Z" level=info msg="POST /api/v1/conversions 200 47.731353ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.186198622Z" level=info msg="received webhook request from 172.19.0.1:52006" module=api
time="2024-01-08T16:06:13.232626827Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:b8bfa6bfda24cf129ce2a20ea3fe679ad377840c0c16d7937887233485b3f170"
time="2024-01-08T16:06:13.279097726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:483bf8eb111365bf322a25443d3f96ae0d80829c60f00fb329d8a0de0f21c6e7"
time="2024-01-08T16:06:13.318753293Z" level=info msg="POST /api/v1/conversions 200 132.527925ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.341869273Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:e870e58e0e1f937652982f99cddff85ab2076d217db08732856d22eb334e9e2a"
time="2024-01-08T16:08:48.548941466Z" level=info msg="converted image anonyme/hub.docker.com/library/mysql:latest-esgz , elapse 2m43.080799299s" module=converter
time="2024-01-08T16:08:48.549005034Z" level=info msg="pushing image anonyme/hub.docker.com/library/mysql:latest-esgz" module=converter
time="2024-01-08T16:08:48.587453917Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:08:48.58782373Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:08:48.597293349Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:08:48.597781217Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:08:48.968420375Z" level=error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"
time="2024-01-08T16:40:47.091302038Z" level=info msg="garbage collect, elapse 8.45699ms"
time="2024-01-08T17:33:13.824434821Z" level=info msg="received webhook request from 172.19.0.1:60738" module=api
time="2024-01-08T17:33:13.914808701Z" level=info msg="POST /api/v1/conversions 200 90.366352ms 617>5bytes 172.19.0.1" module=api
time="2024-01-08T17:33:13.921133726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/sameersbn/apt-cacher-ng@sha256:58e74113cfac7e593201444648c105351cbfce7538bfb36dcafdc9479b2aefcc"
time="2024-01-08T17:39:58.249120243Z" level=info msg="received webhook request from 172.19.0.1:45660" module=api
time="2024-01-08T17:39:58.299701694Z" level=info msg="POST /api/v1/conversions 200 50.5531ms 608>5bytes 172.19.0.1" module=api
time="2024-01-08T17:39:58.30843498Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/gitlab/gitlab-runner@sha256:7a267b16df7d05786fa7d76758e5a0dcc34dc6318902fbfa29aaad7ad3cbe1be"
time="2024-01-09T11:45:17.207698977Z" level=info msg="received webhook request from 172.19.0.1:45606" module=api
time="2024-01-09T11:45:17.223161837Z" level=info msg="POST /api/v1/conversions 200 15.467165ms 596>5bytes 172.19.0.1" module=api
time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"
ptempier commented 8 months ago

I am thinking maybe i need to add some dependencies to acceld ? Right now it run inside a docker made from rockylinux:9, At this moment it's only raw rockylinux, 0 packages added, and accelld copied into /root/

ptempier commented 8 months ago

I added some docker tools inside the container, but it doesn't change anything. level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/alpinelinux/rsyncd@sha256:6f8b68b4b15a8e6b0abfb7db0e2a765849c77a6104ac248810ff9a9fb97996fb"

The Dockerfile i use :

FROM  anonyme/hub.docker.com/library/rockylinux:9

RUN echo "proxy=http://anonyme:3142" >> /etc/yum.conf
RUN dnf -y install dnf-plugins-core && dnf config-manager --set-disabled '*' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/AppStream/$basearch/os/' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/BaseOS/$basearch/os/'

#add the docker part
RUN dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
RUN dnf -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin

WORKDIR "/root"

RUN dnf install -y wget procps
RUN wget "https://raw.githubusercontent.com/goharbor/acceleration-service/main/misc/config/config.estargz.yaml"

RUN wget "https://github.com/goharbor/acceleration-service/releases/download/v0.2.13/harbor-acceld-v0.2.13-linux-amd64.tgz" && tar -xvzf  ./harbor-acceld-v0.2.13-linux-amd64.tgz
RUN ls -lha
RUN chmod ugo+x ./harbor-acceld/acceld ./harbor-acceld/accelctl

RUN  sed -i 's|hub.harbor.com|anonyme|' ./config.estargz.yaml && \
 sed -i 's|auth_header: header|auth_header: anonyme|' ./config.estargz.yaml && \
 sed -i 's|# auth: YTpiCg==|auth: anonyme|' ./config.estargz.yaml

CMD "./harbor-acceld/acceld --config ./config.estargz.yaml"
imeoer commented 8 months ago

@ptempier You seem to be pushing an image with the name format example.com/namespace/repo@sha256:xxx, how about trying to use the name format example.com/namespace/repo:tag instead?

ptempier commented 8 months ago

@imeoer nope i am doing :

docker pull anonyme/hub.docker.com/library/postgres anonyme is the harbor registry server hub.docker.com is the project set as proxy to docker hub library is the default project space in docker hub postgres is the image name my understanding is that docker pull will automatically append the latest is no tag is specfied

So for me it s a very "normal" image pull with a very commonly used registry.

But acceld when receiving such a request respond with : convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

imeoer commented 8 months ago

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

Desiki-high commented 8 months ago

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

Support image@sha265 can be a new feature? @imeoer .

imeoer commented 8 months ago

Support image@sha265 can be a new feature? @imeoer .

@Desiki-high Yes, it should also support the repo@sha256:xxx image name, but a map rule is needed to transform it to the converted image name, possibly repo@sha256:yyy, however, this makes it inconvenient for users to find the converted image.

ptempier commented 8 months ago

Maybe you need a fix on the harbor side to send the proper image name ? The client send the query to harbor asking for image:tag and its harbor who decide that it needs to pull blob in format repo@sha256:xxx then send this to acceld

So maybe a webhook of acceld type is needed.

Desiki-high commented 8 months ago

Maybe you need a fix on the harbor side to send the proper image name ?

The client send the query to harbor asking for image:tag

and its harbor who decide that it needs to pull blob in format repo@sha256:xxx

then send this to acceld

So maybe a webhook of acceld type is needed.

Thanks for your advice! I will ping you if any process.

Desiki-high commented 8 months ago

@ptempier Could you please help to check acceld build from https://github.com/Desiki-high/acceleration-service/tree/feat/image-rule-sha256

ptempier commented 8 months ago

@Desiki-high Thanks, it seems now it can build the estargz but it still can 't push even if it has the permissions.

The robot has the permissions and is authenticated but it get an error from nginx. And that error won't show in the habor logs.

Permissions (Y = ticked) Select all List Repository Y Pull Repository Y Push Repository Y Delete Repository Read Artifact List Artifact Y Delete Artifact Create Artifact label Delete Artifact label Create Tag Y Delete Tag List Tag Y Create Scan Stop Scan

Desiki-high commented 8 months ago

@ptempier I will take a look later.

Desiki-high commented 8 months ago

It looks like a permission error. You can try to confirm your auth with provider.source.xxxxxx.auth in your config yaml. image

ptempier commented 8 months ago

Hello

The auth is fine, the robot appears authenticated when pulling. I double checked inside the container, the token is properly replaced and with the correct indentation.

Desiki-high commented 8 months ago

Hello

The auth is fine, the robot appears authenticated when pulling. I double checked inside the container, the token is properly replaced and with the correct indentation.

I test the new acceld in my harbor registry, it works well with the image like repo@sha256xxx. Any good suggestions? @imeoer

ptempier commented 8 months ago

upgraded habor gave all the permissions to the robot refreshed the token tried witht he habor super admin removed the localhost unauthenticated connection ... still not working

Something strange is the unauthenticated pull in the logs, sometime it s authenticated. Mayeb the code path is wrong and when it tries to push its unauthenticated, i dont know. Couldnt find in the doc if i could get more verbose logs from acceld

ptempier commented 8 months ago

i did re-read the doc, and it says to use a system robot account and not a project robot account, but i get the same issue.

what tickle me here is the call to ldap.go, when its a robot account, but maybe it just search there after checking its not a robot.

2024-01-15T17:49:56Z [ERROR] [/server/middleware/security/robot.go:58][requestID="f567fb86-c90b-42cc-a6f2-d3d1a6599419"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:56Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-01-15T17:49:56Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$hub.docker.com-acceld-estargz, and sleep for 1.5s
2024-01-15T17:49:57Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="172.21.0.1" requestID="f567fb86-c90b-42cc-a6f2-d3d1a6599419" user agent="containerd/1.7.11+unknown"]: failed to authenticate user:robot$hub.docker.com-acceld-estargz, error:Failed to authenticate user, due to error 'Not found an entry'
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="43821675-1d74-49eb-bd28-f946e12a4025"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:58Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-01-15T17:49:58Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$hub.docker.com-acceld-estargz, and sleep for 1.5s
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="ea4db1fa-6856-4f25-91a0-40ca1469520d"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="6452d1d5-21cf-45e4-bc78-3d7121325723"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
... it does about 100 tries
oliverbaehler commented 8 months ago

@ptempier Thanks for opening the issue, i came across around the same behavior. Mind sharing your config.yaml? But if I understand correctly it should not be possible to push anything to a project with docker proxy behind (That's also stated in the harbor doc). You also get a permission denied when trying to push anything :

The push refers to repository [harbor/dockerhub/test]
81150088de4c: Preparing
c909727f9cc1: Preparing
4f4ce317c6bb: Preparing
denied: denied: can not push artifact to a proxy project: dockerhub

/dockerhub points to hub.docker.com. Since your logs indicate you only have a 403 on image pushes, i suspect that's what's happening.

I suspect you would need to consider pushing to a different project, which is not a docker proxy. If i am not completely misunderstanding this thread

ptempier commented 8 months ago

@oliverbaehler That's the question i asked first, but apparently its supposed to work. @Desiki-high says its working for him, but maybe its not well tested and works on in the specific setup of some testbed.

Desiki-high commented 8 months ago

@oliverbaehler That's the question i asked first, but apparently its supposed to work.

@Desiki-high says its working for him, but maybe its not well tested and works on in the specific setup of some testbed.

Sorry, my mistake. I did the test without setting up the proxy. I just fix the issue with image format.

oliverbaehler commented 8 months ago

@Desiki-high The question is, if we could figure out, if a image is from a docker proxy registry. If so, we need to push it to a different project (idk if we could do any rules). @ptempier What you are trying is currently not supported. Although I was also thinking about that same use case. But you would need as mentioned to push the tags to a different registry and then work with multiple mirrors per registry.

Or the upstream projects release estargz or nydus images..