Closed NexZhu closed 1 year ago
Which harbor version? It seems to work fine in our e2e tests using Harbor 2.7.0: https://github.com/docker/build-push-action/actions/runs/5783700939/job/15673017675
I'm testing with v2.8.3
Some more details:
I've tested docker push
to our Harbor registry with the same credential (~/.docker/config.json`) on another server and it's working fine.
We can see from the Traefik access log that Docker client is calling GET /service/token
endpoint with an Authorization
header, which is the value of the auth
field in ~/.docker/config.json
.
{
"ClientPort": "34114",
"ClientUsername": "-",
"DownstreamContentSize": 716,
"DownstreamStatus": 200,
"Duration": 16820030,
"OriginContentSize": 716,
"OriginDuration": 16779299,
"OriginStatus": 200,
"Overhead": 40731,
"RequestContentSize": 0,
"RequestCount": 409,
"RequestMethod": "GET",
"RequestPath": "/service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry",
"RequestPort": "-",
"RequestProtocol": "HTTP/1.1",
"RequestScheme": "https",
"RetryAttempts": 0,
"ServiceAddr": "172.16.15.41:8080",
"ServiceName": "harbor-harbor-core-80@kubernetes",
"ServiceURL": "http://172.16.15.41:8080",
"StartLocal": "2023-08-07T03:00:37.534525546Z",
"StartUTC": "2023-08-07T03:00:37.534525546Z",
"TLSCipher": "TLS_AES_128_GCM_SHA256",
"TLSVersion": "1.3",
"downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"downstream_Content-Encoding": "gzip",
"downstream_Content-Length": "716",
"downstream_Content-Type": "application/json; charset=utf-8",
"downstream_Date": "Mon, 07 Aug 2023 03:00:37 GMT",
"downstream_Set-Cookie": "sid=08aad51a8c1d5dedde2f45feb28f00c7; Path=/; HttpOnly",
"downstream_X-Request-Id": "c8019aec-b58b-49a6-95bc-8d24547bcfb9",
"entryPointName": "websecure",
"level": "info",
"msg": "",
"origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"origin_Content-Encoding": "gzip",
"origin_Content-Length": "716",
"origin_Content-Type": "application/json; charset=utf-8",
"origin_Date": "Mon, 07 Aug 2023 03:00:37 GMT",
"origin_Set-Cookie": "sid=08aad51a8c1d5dedde2f45feb28f00c7; Path=/; HttpOnly",
"origin_X-Request-Id": "c8019aec-b58b-49a6-95bc-8d24547bcfb9",
"request_Accept-Encoding": "gzip",
"request_Authorization": "Basic xxxxxx (censored)", <-- Authorization header here, `xxxxxx` is the value of the `auth` field in `~/.docker/config.json`
"request_Connection": "close",
"request_User-Agent": "docker/20.10.15 go/go1.17.9 git-commit/4433bf6 kernel/4.18.0-383.el8.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.15 \\(linux\\))",
"request_X-Forwarded-Port": "443",
"request_X-Forwarded-Proto": "https",
"request_X-Forwarded-Server": "traefik-kjg2b",
"time": "2023-08-07T03:00:37Z"
}
In Harbor log, we can see a robot security context generated for request GET /service/token
which means auth suceeded:
2023-08-05T02:35:38Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0c81a4ae-611f-49f0-b02f-287e63520b08 to the logger for the request GET /service/token
2023-08-05T02:35:38Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry
2023-08-05T02:35:38Z [INFO] [/server/middleware/security/robot.go:71][requestID="0c81a4ae-611f-49f0-b02f-287e63520b08"]: a robot security context generated for request GET /service/token
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?account=robot%24k8s&scope=repository%3Afgfe%2Ffes-starter%3Apush%2Cpull&service=harbor-registry
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/creator.go:231]: scopes: [repository:fgfe/fes-starter:push,pull]
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:fgfe/fes-starter:push,pull]
2023/08/05 02:35:38 Model:
2023/08/05 02:35:38 r.r: sub, obj, act
2023/08/05 02:35:38 p.p: sub, obj, act, eft
2023/08/05 02:35:38 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2023/08/05 02:35:38 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2023/08/05 02:35:38 g.g: _, _
2023/08/05 02:35:38 Policy:
2023/08/05 02:35:38 p: sub, obj, act, eft: [[robot$k8s /project/*/repository list allow] [robot$k8s /project/*/repository pull allow] [robot$k8s /project/*/repository push allow] [robot$k8s /project/*/repository delete allow] [robot$k8s /project/*/artifact read allow] [robot$k8s /project/*/artifact list allow] [robot$k8s /project/*/artifact delete allow] [robot$k8s /project/*/artifact-label create allow] [robot$k8s /project/*/artifact-label delete allow] [robot$k8s /project/*/tag create allow] [robot$k8s /project/*/tag delete allow] [robot$k8s /project/*/tag list allow] [robot$k8s /project/*/scan create allow] [robot$k8s /project/*/scan stop allow]]
2023/08/05 02:35:38 g: _, _: []
2023/08/05 02:35:38 Role links for: g
2023/08/05 02:35:38
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, delete ---> true
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, scanner-pull ---> false
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, pull ---> true
2023/08/05 02:35:38 Request: robot$k8s, /project/9/repository, push ---> true
2023-08-05T02:35:38Z [DEBUG] [/core/service/token/authutils.go:102]: user: robot$k8s, access: &{repository fgfe/fes-starter [delete pull push *]}
But with docker/build-push-action
and the same harbor account, pushing failed with 401 Unauthorized
.
I checked the Traefik access log and the action was not sending Authorization
header when requesting the /server/token
endpoint:
{
"ClientPort": "34256",
"ClientUsername": "-",
"DownstreamContentSize": 674,
"DownstreamStatus": 200,
"Duration": 5541774,
"OriginContentSize": 674,
"OriginDuration": 5508937,
"OriginStatus": 200,
"Overhead": 32837,
"RequestContentSize": 0,
"RequestCount": 2743,
"RequestMethod": "GET",
"RequestPath": "/service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry",
"RequestPort": "-",
"RequestProtocol": "HTTP/2.0",
"RequestScheme": "https",
"RetryAttempts": 0,
"ServiceAddr": "172.16.15.41:8080",
"ServiceName": "harbor-harbor-core-80@kubernetes",
"ServiceURL": "http://172.16.15.41:8080",
"StartLocal": "2023-08-07T03:39:39.477955336Z",
"StartUTC": "2023-08-07T03:39:39.477955336Z",
"TLSCipher": "TLS_AES_128_GCM_SHA256",
"TLSVersion": "1.3",
"downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"downstream_Content-Encoding": "gzip",
"downstream_Content-Length": "674",
"downstream_Content-Type": "application/json; charset=utf-8",
"downstream_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
"downstream_Set-Cookie": "sid=65eb89b1d1b5e52ce08cef0c4a3d756a; Path=/; HttpOnly",
"downstream_X-Request-Id": "8dcbf311-21af-49cf-b3bf-8612eb4339ba",
"entryPointName": "websecure",
"level": "info",
"msg": "",
"origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"origin_Content-Encoding": "gzip",
"origin_Content-Length": "674",
"origin_Content-Type": "application/json; charset=utf-8",
"origin_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
"origin_Set-Cookie": "sid=65eb89b1d1b5e52ce08cef0c4a3d756a; Path=/; HttpOnly",
"origin_X-Request-Id": "8dcbf311-21af-49cf-b3bf-8612eb4339ba",
"request_Accept-Encoding": "gzip",
"request_User-Agent": "containerd/1.7.2+unknown",
"request_X-Forwarded-Port": "443",
"request_X-Forwarded-Proto": "https",
"request_X-Forwarded-Server": "traefik-kjg2b",
"time": "2023-08-07T03:39:39Z"
}
In Harbor log, we can see an unauthorized security context generated for request GET /service/token
, that's because the Authorization
header was missing, so we got a unauthorized security context
:
2023-08-05T02:32:05Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 22ee7597-fd52-4ad8-bf74-b65a5dd0acb7 to the logger for the request GET /service/token
2023-08-05T02:32:05Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry
2023-08-05T02:32:05Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="22ee7597-fd52-4ad8-bf74-b65a5dd0acb7"]: an unauthorized security context generated for request GET /service/token
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?scope=repository%3Afgfe%2Ffes-starter%3Apull%2Cpush&service=harbor-registry
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/creator.go:231]: scopes: [repository:fgfe/fes-starter:pull,push]
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:fgfe/fes-starter:pull,push]
2023/08/05 02:32:05 Model:
2023/08/05 02:32:05 r.r: sub, obj, act
2023/08/05 02:32:05 p.p: sub, obj, act, eft
2023/08/05 02:32:05 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2023/08/05 02:32:05 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2023/08/05 02:32:05 g.g: _, _
2023/08/05 02:32:05 Policy:
2023/08/05 02:32:05 p: sub, obj, act, eft: []
2023/08/05 02:32:05 g: _, _: []
2023/08/05 02:32:05 Role links for: g
2023/08/05 02:32:05
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, delete ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, scanner-pull ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, pull ---> false
2023/08/05 02:32:05 Request: anonymous, /project/9/repository, push ---> false
2023-08-05T02:32:05Z [DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository fgfe/fes-starter []}
Which resulted in the action not getting the authorized token, and the next request got 401 Unauthorized
as a result:
{
"ClientPort": "55008",
"ClientUsername": "-",
"DownstreamContentSize": 0,
"DownstreamStatus": 401, <--
"Duration": 2632666,
"OriginContentSize": 0,
"OriginDuration": 2594614,
"OriginStatus": 401, <--
"Overhead": 38052,
"RequestContentSize": 0,
"RequestCount": 2747,
"RequestMethod": "HEAD",
"RequestPath": "/v2/fgfe/fes-starter/blobs/sha256:7a83023678a94edc80ac36165fcd867535fa82bac546fb2c83e019d67189d7d7",
"RequestPort": "-",
"RequestProtocol": "HTTP/1.1",
"RequestScheme": "https",
"RetryAttempts": 0,
"ServiceAddr": "172.16.15.41:8080",
"ServiceName": "harbor-harbor-core-80@kubernetes",
"ServiceURL": "http://172.16.15.41:8080",
"StartLocal": "2023-08-07T03:39:39.485499055Z",
"StartUTC": "2023-08-07T03:39:39.485499055Z",
"TLSCipher": "TLS_AES_128_GCM_SHA256",
"TLSVersion": "1.3",
"downstream_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"downstream_Content-Length": "182",
"downstream_Content-Type": "application/json; charset=utf-8",
"downstream_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
"downstream_Docker-Distribution-Api-Version": "registry/2.0",
"downstream_Set-Cookie": "sid=6aebe238b4d512fde6ba4ea8b345ff7e; Path=/; HttpOnly",
"downstream_Www-Authenticate": "Basic realm=\"harbor\"",
"downstream_X-Request-Id": "13544d1a-9fc9-48f0-a04c-816ee41cbaa8",
"entryPointName": "websecure",
"level": "info",
"msg": "",
"origin_Alt-Svc": "h3=\":8443\"; ma=2592000,h3-29=\":8443\"; ma=2592000",
"origin_Content-Length": "182",
"origin_Content-Type": "application/json; charset=utf-8",
"origin_Date": "Mon, 07 Aug 2023 03:39:39 GMT",
"origin_Docker-Distribution-Api-Version": "registry/2.0",
"origin_Set-Cookie": "sid=6aebe238b4d512fde6ba4ea8b345ff7e; Path=/; HttpOnly",
"origin_Www-Authenticate": "Basic realm=\"harbor\"",
"origin_X-Request-Id": "13544d1a-9fc9-48f0-a04c-816ee41cbaa8",
"request_Accept": "application/vnd.oci.image.layer.v1.tar+gzip, */*",
"request_Authorization": "Bearer xxxxxx",
"request_Traceparent": "00-e3b525613eccd7bd48c55cbc19e1ce92-caaaac4b51e7e8bd-01",
"request_User-Agent": "buildkit/v0.12",
"request_X-Forwarded-Port": "443",
"request_X-Forwarded-Proto": "https",
"request_X-Forwarded-Server": "traefik-kjg2b",
"time": "2023-08-07T03:39:39Z"
}
Looks like you already opened an issue on BuildKit repo: https://github.com/moby/buildkit/issues/4111
Did you try to repro locally? I see you're using a remote BuildKit instance. If it doesn't work locally as well, let's continue the convo in the BuildKit issue.
@crazy-max Yes, I thought the issue was related to BuildKit at first, but I was told that the credential is set from the client (Docker), not the server (buildkit) https://github.com/moby/buildkit/issues/4111#issuecomment-1668776883, so I posted here instead. I'm building in K8s with containerd instead of Docker (meaning I cannot use dind), so I can only build with remote driver with buildkit.
Sure but do you repro locally or not?
Indeed, I can repro locally, docker push
worked, but docker buildx --push
failed with the same 401 Unauthorized
, I'm closing the issue here, thanks!
Behaviour
Steps to reproduce this issue
I'm using Gitea Actions, my
build.yaml
:docker/login-action@v2
and build both succeeded, but pushing failed with401 Unauthorized
I'm sure the credential I'm using has the push right, because I've tested with the same credential on a server manually
docker login
anddocker push
to the same repository, it worked.Expected behaviour
Should be able to push to Harbor.