search

docker / build-push-action

GitHub Action to build and push Docker images with Buildx
https://github.com/marketplace/actions/build-and-push-docker-images
Apache License 2.0
4.24k stars 541 forks source link

Push to Harbor failed with `401 Unauthorized` despite `docker/login-action@v2` suceeded #923

Closed NexZhu closed 1 year ago

NexZhu commented 1 year ago

Behaviour

Steps to reproduce this issue

I'm using Gitea Actions, mybuild.yaml:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver: remote
          endpoint: tcp://buildkitd.buildkit.svc.cluster.local:1234

      - uses: docker/login-action@v2
        with:
          registry: harbor.internetapi.cn
          username: ${{ secrets.HARBOR_USERNAME }}
          password: ${{ secrets.HARBOR_PASSWORD }}

      - uses: docker/build-push-action@master
        with:
          context: .
          file: docker/Dockerfile.prebuilt
          push: true
          tags: harbor.redacted.com/fgfe/fes-starter:latest

docker/login-action@v2 and build both succeeded, but pushing failed with 401 Unauthorized

image image

I'm sure the credential I'm using has the push right, because I've tested with the same credential on a server manually docker login and docker push to the same repository, it worked.

Expected behaviour

Should be able to push to Harbor.

crazy-max commented 1 year ago

Which harbor version? It seems to work fine in our e2e tests using Harbor 2.7.0: https://github.com/docker/build-push-action/actions/runs/5783700939/job/15673017675

NexZhu commented 1 year ago

I'm testing with v2.8.3

NexZhu commented 1 year ago

Some more details:

I've tested docker push to our Harbor registry with the same credential (~/.docker/config.json`) on another server and it's working fine.

We can see from the Traefik access log that Docker client is calling GET /service/token endpoint with an Authorization header, which is the value of the auth field in ~/.docker/config.json.

{
    "ClientPort": "34114",
    "ClientUsername": "-",
    "DownstreamContentSize": 716,
    "DownstreamStatus": 200,
    "Duration": 16820030,
    "OriginContentSize": 716,
    "OriginDuration": 16779299,
    "OriginStatus": 200,
    "Overhead": 40731,
    "RequestContentSize": 0,
    "RequestCount": 409,
    "RequestMethod": "GET",
    "RequestPath": "/service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "ServiceAddr": "172.16.15.41:8080",
    "ServiceName": "harbor-harbor-core-80@kubernetes",
    "ServiceURL": "http://172.16.15.41:8080",
    "StartLocal": "2023-08-07T03:00:37.534525546Z",
    "StartUTC": "2023-08-07T03:00:37.534525546Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "downstream_Content-Encoding": "gzip",
    "downstream_Content-Length": "716",
    "downstream_Content-Type": "application/json; charset=utf-8",
    "downstream_Date": "Mon, 07 Aug 2023 03:00:37 GMT",
    "downstream_Set-Cookie": "sid=08aad51a8c1d5dedde2f45feb28f00c7; Path=/; HttpOnly",
    "downstream_X-Request-Id": "c8019aec-b58b-49a6-95bc-8d24547bcfb9",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "origin_Content-Encoding": "gzip",
    "origin_Content-Length": "716",
    "origin_Content-Type": "application/json; charset=utf-8",
    "origin_Date": "Mon, 07 Aug 2023 03:00:37 GMT",
    "origin_Set-Cookie": "sid=08aad51a8c1d5dedde2f45feb28f00c7; Path=/; HttpOnly",
    "origin_X-Request-Id": "c8019aec-b58b-49a6-95bc-8d24547bcfb9",
    "request_Accept-Encoding": "gzip",
    "request_Authorization": "Basic xxxxxx (censored)", <-- Authorization header here, `xxxxxx` is the value of the `auth` field in `~/.docker/config.json`
    "request_Connection": "close",
    "request_User-Agent": "docker/20.10.15 go/go1.17.9 git-commit/4433bf6 kernel/4.18.0-383.el8.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.15 \\(linux\\))",
    "request_X-Forwarded-Port": "443",
    "request_X-Forwarded-Proto": "https",
    "request_X-Forwarded-Server": "traefik-kjg2b",
    "time": "2023-08-07T03:00:37Z"
}

In Harbor log, we can see a robot security context generated for request GET /service/token which means auth suceeded:

2023-08-05T02:35:38Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0c81a4ae-611f-49f0-b02f-287e63520b08 to the logger for the request GET /service/token
2023-08-05T02:35:38Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry
2023-08-05T02:35:38Z [INFO] [/server/middleware/security/robot.go:71][requestID="0c81a4ae-611f-49f0-b02f-287e63520b08"]: a robot security context generated for request GET /service/token
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/creator.go:231]: scopes: [repository:fgfe/fes-starter:push,pull]
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:fgfe/fes-starter:push,pull]
2023/08/05 02:35:38 Model:
2023/08/05 02:35:38 r.r: sub, obj, act
2023/08/05 02:35:38 p.p: sub, obj, act, eft
2023/08/05 02:35:38 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2023/08/05 02:35:38 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2023/08/05 02:35:38 g.g: _, _
2023/08/05 02:35:38 Policy:
2023/08/05 02:35:38 p: sub, obj, act, eft: [[robot$k8s /project/*/repository list allow] [robot$k8s /project/*/repository pull allow] [robot$k8s /project/*/repository push allow] [robot$k8s /project/*/repository delete allow] [robot$k8s /project/*/artifact read allow] [robot$k8s /project/*/artifact list allow] [robot$k8s /project/*/artifact delete allow] [robot$k8s /project/*/artifact-label create allow] [robot$k8s /project/*/artifact-label delete allow] [robot$k8s /project/*/tag create allow] [robot$k8s /project/*/tag delete allow] [robot$k8s /project/*/tag list allow] [robot$k8s /project/*/scan create allow] [robot$k8s /project/*/scan stop allow]]
2023/08/05 02:35:38 g: _, _: []
2023/08/05 02:35:38 Role links for: g
2023/08/05 02:35:38 
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, delete ---> true
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, scanner-pull ---> false
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, pull ---> true
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, push ---> true
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/authutils.go:102]: user: robot$k8s, access: &{repository  fgfe/fes-starter [delete pull push *]}

But with docker/build-push-action and the same harbor account, pushing failed with 401 Unauthorized.

I checked the Traefik access log and the action was not sending Authorization header when requesting the /server/token endpoint:

{
    "ClientPort": "34256",
    "ClientUsername": "-",
    "DownstreamContentSize": 674,
    "DownstreamStatus": 200,
    "Duration": 5541774,
    "OriginContentSize": 674,
    "OriginDuration": 5508937,
    "OriginStatus": 200,
    "Overhead": 32837,
    "RequestContentSize": 0,
    "RequestCount": 2743,
    "RequestMethod": "GET",
    "RequestPath": "/service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/2.0",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "ServiceAddr": "172.16.15.41:8080",
    "ServiceName": "harbor-harbor-core-80@kubernetes",
    "ServiceURL": "http://172.16.15.41:8080",
    "StartLocal": "2023-08-07T03:39:39.477955336Z",
    "StartUTC": "2023-08-07T03:39:39.477955336Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "downstream_Content-Encoding": "gzip",
    "downstream_Content-Length": "674",
    "downstream_Content-Type": "application/json; charset=utf-8",
    "downstream_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
    "downstream_Set-Cookie": "sid=65eb89b1d1b5e52ce08cef0c4a3d756a; Path=/; HttpOnly",
    "downstream_X-Request-Id": "8dcbf311-21af-49cf-b3bf-8612eb4339ba",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "origin_Content-Encoding": "gzip",
    "origin_Content-Length": "674",
    "origin_Content-Type": "application/json; charset=utf-8",
    "origin_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
    "origin_Set-Cookie": "sid=65eb89b1d1b5e52ce08cef0c4a3d756a; Path=/; HttpOnly",
    "origin_X-Request-Id": "8dcbf311-21af-49cf-b3bf-8612eb4339ba",
    "request_Accept-Encoding": "gzip",
    "request_User-Agent": "containerd/1.7.2+unknown",
    "request_X-Forwarded-Port": "443",
    "request_X-Forwarded-Proto": "https",
    "request_X-Forwarded-Server": "traefik-kjg2b",
    "time": "2023-08-07T03:39:39Z"
}

In Harbor log, we can see an unauthorized security context generated for request GET /service/token, that's because the Authorization header was missing, so we got a unauthorized security context:

2023-08-05T02:32:05Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 22ee7597-fd52-4ad8-bf74-b65a5dd0acb7 to the logger for the request GET /service/token
2023-08-05T02:32:05Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry
2023-08-05T02:32:05Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="22ee7597-fd52-4ad8-bf74-b65a5dd0acb7"]: an unauthorized security context generated for request GET /service/token
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/creator.go:231]: scopes: [repository:fgfe/fes-starter:pull,push]
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:fgfe/fes-starter:pull,push]
2023/08/05 02:32:05 Model:
2023/08/05 02:32:05 r.r: sub, obj, act
2023/08/05 02:32:05 p.p: sub, obj, act, eft
2023/08/05 02:32:05 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2023/08/05 02:32:05 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2023/08/05 02:32:05 g.g: _, _
2023/08/05 02:32:05 Policy:
2023/08/05 02:32:05 p: sub, obj, act, eft: []
2023/08/05 02:32:05 g: _, _: []
2023/08/05 02:32:05 Role links for: g
2023/08/05 02:32:05 
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, delete ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, scanner-pull ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, pull ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, push ---> false
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  fgfe/fes-starter []}

Which resulted in the action not getting the authorized token, and the next request got 401 Unauthorized as a result:

{
    "ClientPort": "55008",
    "ClientUsername": "-",
    "DownstreamContentSize": 0,
    "DownstreamStatus": 401, <--
    "Duration": 2632666,
    "OriginContentSize": 0,
    "OriginDuration": 2594614,
    "OriginStatus": 401, <--
    "Overhead": 38052,
    "RequestContentSize": 0,
    "RequestCount": 2747,
    "RequestMethod": "HEAD",
    "RequestPath": "/v2/fgfe/fes-starter/blobs/sha256:7a83023678a94edc80ac36165fcd867535fa82bac546fb2c83e019d67189d7d7",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "ServiceAddr": "172.16.15.41:8080",
    "ServiceName": "harbor-harbor-core-80@kubernetes",
    "ServiceURL": "http://172.16.15.41:8080",
    "StartLocal": "2023-08-07T03:39:39.485499055Z",
    "StartUTC": "2023-08-07T03:39:39.485499055Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "downstream_Content-Length": "182",
    "downstream_Content-Type": "application/json; charset=utf-8",
    "downstream_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
    "downstream_Docker-Distribution-Api-Version": "registry/2.0",
    "downstream_Set-Cookie": "sid=6aebe238b4d512fde6ba4ea8b345ff7e; Path=/; HttpOnly",
    "downstream_Www-Authenticate": "Basic realm=\"harbor\"",
    "downstream_X-Request-Id": "13544d1a-9fc9-48f0-a04c-816ee41cbaa8",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
    "origin_Content-Length": "182",
    "origin_Content-Type": "application/json; charset=utf-8",
    "origin_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
    "origin_Docker-Distribution-Api-Version": "registry/2.0",
    "origin_Set-Cookie": "sid=6aebe238b4d512fde6ba4ea8b345ff7e; Path=/; HttpOnly",
    "origin_Www-Authenticate": "Basic realm=\"harbor\"",
    "origin_X-Request-Id": "13544d1a-9fc9-48f0-a04c-816ee41cbaa8",
    "request_Accept": "application/vnd.oci.image.layer.v1.tar+gzip, */*",
    "request_Authorization": "Bearer xxxxxx",
    "request_Traceparent": "00-e3b525613eccd7bd48c55cbc19e1ce92-caaaac4b51e7e8bd-01",
    "request_User-Agent": "buildkit/v0.12",
    "request_X-Forwarded-Port": "443",
    "request_X-Forwarded-Proto": "https",
    "request_X-Forwarded-Server": "traefik-kjg2b",
    "time": "2023-08-07T03:39:39Z"
}
crazy-max commented 1 year ago

Looks like you already opened an issue on BuildKit repo: https://github.com/moby/buildkit/issues/4111

Did you try to repro locally? I see you're using a remote BuildKit instance. If it doesn't work locally as well, let's continue the convo in the BuildKit issue.

NexZhu commented 1 year ago

@crazy-max Yes, I thought the issue was related to BuildKit at first, but I was told that the credential is set from the client (Docker), not the server (buildkit) https://github.com/moby/buildkit/issues/4111#issuecomment-1668776883, so I posted here instead. I'm building in K8s with containerd instead of Docker (meaning I cannot use dind), so I can only build with remote driver with buildkit.

crazy-max commented 1 year ago

Sure but do you repro locally or not?

NexZhu commented 1 year ago

Indeed, I can repro locally, docker push worked, but docker buildx --push failed with the same 401 Unauthorized, I'm closing the issue here, thanks!