Using personal access token with Dockerhub API Beta

[qbiqing]

Problem description

When trying to use the Dockerhub beta API, such as this endpoint: /v2/namespaces/{namespace}/repositories/{repository}/images,

I used the authentication method through Personal Access Tokens as introduced in the docs. That is, I created a JWT using the authentication endpoint in this section and with the personal access token instead of the password in the password field. I was able to get a JWT in return, however, when I use it to get details about the images, I got this error:

{
    "message": "access to the resource is forbidden with personal access token"
}

I'm confused if the personal access token is an acceptable mode of authentication? This is not clear in the docs.

[mikeparker]

Apologies, that sounds like a gap in the docs. PATs are not currently supported for API operations, but this is something we are looking to implement in future.

[qbiqing]

@mikeparker thanks, good to know! It would be really useful for apps using the Dockerhub API.

[qbiqing]

To add on, the docs now are even more misleading, because this is under the section of "Create an authentication token": It says PAT can be used to authenticate with but I still get forbidden access to resource.

[mikeparker]

@qbiqing you're absolutely right, the docs are misleading here. I'll get this fixed. Sorry again.

[modem7]

Apologies, that sounds like a gap in the docs. PATs are not currently supported for API operations, but this is something we are looking to implement in future.

@mikeparker I thought that https://github.com/docker/roadmap/issues/115 was supposed to have been rolled out for this very purpose?

[modem7]

Is there any updates on this?

There doesn't seem to be much of a point to 2fa if you're unable to upload something as simple as a readme file.

[srbala]

PATs are not currently supported for API operations, but this is something we are looking to implement in future. @mikeparker Any tentative timeline for this to be implemented

[github-actions[bot]]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[modem7]

Stalebot bump

[technicallyjosh]

I think something to help this confusion on the error message is to return something like "access to the resource is forbidden with an access token generated from a PAT". As far as the API access, this was bumped my way and should be fairly simple to patch in since it is a repo scoped call. (Right now PATs only have repo scopes). We can definitely look into allowing for PAT generated JWTs to access this resource with the repo:read scope.

[technicallyjosh]

I've opened a PR to allow for image management route usage with PAT generated JWTs in our APIs. We can address the verbiage on the 403 this week.

[technicallyjosh]

👋🏼 You can now use a properly scoped JWT generated from a PAT on these image management routes. Changes to the wording should be updated within the next week.

[technicallyjosh]

I'll close this out. Thank you for leaving the feedback! Sorry it took a while to get to!

[modem7]

👋🏼 You can now use a properly scoped JWT generated from a PAT on these image management routes. Changes to the wording should be updated within the next week.

Heya @technicallyjosh

Lots of people are still having issues with this.

Do you have a link to the appropriate documentation to get this to work as intended?

Thanks!

[christian-korneck]

@technicallyjosh Thanks for this. Unfortunately I don't seem to be able to get this to work.

What I try is:

• generate JWT for PAT (GET /v2/users/login) - works
  • note: the PAT has repo:admin scope
• try to update a repo description (use JWT as bearer token for PATCH /v2/repositories/my-user/my-repo/) - fails

Result is a 403 FORBIDDEN with message:

access is forbidden with a JWT issued from a personal access token

Am I doing anything incorrectly? Would appreciate any pointer into the right direction. Thanks a lot in advance!

[technicallyjosh]

@modem7 @christian-korneck Hey thanks for the ping on this not working still. It seems we might have some sort of regression here. I do know we've been slaying tech debt and moving APIs around specifically with registry/repos, so I'm wondering if something got lost in translation behind the scenes. I'll have some bandwidth to look at this this weekend and early next week and keep you updated.

@modem7 Because of some of these services being under refactoring, etc, we've been very selective as to what we publicly document. I'll ping the team and see if they are comfortable with documenting these APIs in the given state.

[technicallyjosh]

Okay so it seems like there were some routes missed such as the one you mentioned: PATCH /v2/repositories/{namespace}/{repo}. I'll put a PR in to see if I can get the rest of these to work with the tokens.

Edit: I have a PR that should take care of any missing repository management routes allowing for PAT usage there. I'll still discuss with the team what we want to do about public API docs.

[technicallyjosh]

👋🏼 These fixes are now live on production. Let us know if there are any issues 😄. I can work with the team on updating the proper docs for these. If your PAT doesn't have the right scopes, you will receive a message with something like "insufficient scope" or something of the like. If you get that, you may need to tweak the scopes for the token used to generate the JWT.

[christian-korneck]

@technicallyjosh thanks a lot for looking into this so quickly, much appreciated!

Unfortunately I still don't seem to get my use case to be working. Here's what I do:

1.) create a PAT via the DockerHub WebUI with read, write, delete scope. 2.) generate JWT. The password is a personal access token.

curl --location --request POST 'https://hub.docker.com/v2/users/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
  "username": "my-docker-username",
  "password": "00000000-0000-0000-0000-000000000000"
}'

the result body contains a JWT in the style of

{"token":"ey...pg"}}

3.) Use the token in a request to update a repo's description:

curl --location --request PATCH 'https://hub.docker.com/v2/repositories/my-docker-user/my-docker-repo/' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer ey...pg' \
--data-raw '{
  "description": "Hello World!"
}'

The result is a 403 FORBIDDEN:

{
    "detail": "access is forbidden with a JWT issued from a personal access token"
}

(The same example works if I use a password to create the JWT instead of a PAT)

Do you have any idea what I might be doing wrong? I highly appreciate any pointer into the right direction. (I assume it must be some stupid mistake on my side)?

Again, thanks a lot for your time and efforts!!

[technicallyjosh]

Of course! The JWT generation is fine.

Interesting... I'll have to look into that as I know I allowed the PATCH to be used with a PAT generated token. It's possible that that route is going to a service that I was not aware of (we are migrating from an older service and it may still be used there).

[modem7]

Of course! The JWT generation is fine.

Interesting... I'll have to look into that as I know I allowed the PATCH to be used with a PAT generated token. It's possible that that route is going to a service that I was not aware of (we are migrating from an older service and it may still be used there).

Heya @technicallyjosh,

Would it make sense keeping this issue open until we can do community verification that all works as expected as this is still not working as expected.

[zigarn]

Hi @technicallyjosh, any news about the problem raised by @christian-korneck (https://github.com/docker/hub-feedback/issues/2127#issuecomment-1155664799) that is still existant. It is generating failure in CI jobs trying to update repo's description.

[zhangguanzhang]

@technicallyjosh thanks a lot for looking into this so quickly, much appreciated!

Unfortunately I still don't seem to get my use case to be working. Here's what I do:

1.) create a PAT via the DockerHub WebUI with read, write, delete scope. 2.) generate JWT. The password is a personal access token.

curl --location --request POST 'https://hub.docker.com/v2/users/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
  "username": "my-docker-username",
  "password": "00000000-0000-0000-0000-000000000000"
}'

the result body contains a JWT in the style of

{"token":"ey...pg"}}

3.) Use the token in a request to update a repo's description:

curl --location --request PATCH 'https://hub.docker.com/v2/repositories/my-docker-user/my-docker-repo/' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer ey...pg' \
--data-raw '{
  "description": "Hello World!"
}'

The result is a 403 FORBIDDEN:

{
    "detail": "access is forbidden with a JWT issued from a personal access token"
}

(The same example works if I use a password to create the JWT instead of a PAT)

Do you have any idea what I might be doing wrong? I highly appreciate any pointer into the right direction. (I assume it must be some stupid mistake on my side)?

Again, thanks a lot for your time and efforts!!

same issue @milosgajdos PATL 👀

[zappy-shu]

Good afternoon @zhangguanzhang

We are rolling out personal access token support across endpoints as we speak. The PATCH repository endpoint was rolled out today with support for admin scoped PATs. Could you ensure your PAT has admin scope and test again for me?

Thanks

[zhangguanzhang]

Good afternoon @zhangguanzhang

We are rolling out personal access token support across endpoints as we speak. The PATCH repository endpoint was rolled out today with support for admin scoped PATs. Could you ensure your PAT has admin scope and test again for me?

Thanks

I used the Read, Write, Delete, but does not work for me.

repo_user=zhangguanzhang
repo_token=xxx-xxx-xxx-xxx-xxx
token=$(curl -s -H "Content-Type: application/json" -X POST \
    -d '{"username": "'${repo_user}'", "password": "'${repo_token}'"}' \
    https://hub.docker.com/v2/users/login/ | jq -r .token)

$ curl -iLH "Authorization: Bearer ${token}" -X DELETE \
    https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test
HTTP/1.1 301 MOVED PERMANENTLY
date: Wed, 13 Jul 2022 12:05:09 GMT
content-type: text/html; charset=utf-8
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657713969
x-ratelimit-remaining: 599
x-frame-options: deny
location: https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test/
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

HTTP/1.1 403 FORBIDDEN
date: Wed, 13 Jul 2022 12:05:10 GMT
content-type: application/json
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657713970
x-ratelimit-remaining: 598
x-frame-options: deny
allow: GET, DELETE, HEAD, OPTIONS
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

{"detail": "access is forbidden with a JWT issued from a personal access token"}

or used the jwt

$ curl -iLH "Authorization: JWT ${token}" -X DELETE \
    https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test
HTTP/1.1 301 MOVED PERMANENTLY
date: Wed, 13 Jul 2022 12:05:46 GMT
content-type: text/html; charset=utf-8
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657714006
x-ratelimit-remaining: 597
x-frame-options: deny
location: https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test/
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

HTTP/1.1 403 FORBIDDEN
date: Wed, 13 Jul 2022 12:05:46 GMT
content-type: application/json
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657714006
x-ratelimit-remaining: 596
x-frame-options: deny
allow: GET, DELETE, HEAD, OPTIONS
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

{"detail": "access is forbidden with a JWT issued from a personal access token"}

if I used the password for repo_token, It works

[zappy-shu]

@zhangguanzhang could you do a verbose curl and list the response headers for me? And give me the time/date that you make the call. Thank you

[zhangguanzhang]

@zhangguanzhang could you do a verbose curl and list the response headers for me? And give me the time/date that you make the call. Thank you

I updated the previous comment with more details

[zappy-shu]

@zhangguanzhang that appears to be a DELETE tag operation rather than a PATCH repo. Deleting a tag has not yet been migrated over so does not currently have PAT support (it's next on the list). Does PATCH repo work for you now?

[zhangguanzhang]

@zhangguanzhang that appears to be a DELETE tag operation rather than a PATCH repo. Deleting a tag has not yet been migrated over so does not currently have PAT support (it's next on the list). Does PATCH repo work for you now?

How to delete a tag use the api 😢,I just want to delete a tag

[zigarn]

My test executing the code from https://circleci.com/developer/orbs/orb/circleci/docker#commands-update-description to update description:

JWT=$(curl -s -d "username=$DOCKERHUB_USERNAME&password=$DOCKERHUB_PAT" https://hub.docker.com/v2/users/login/ | jq -r .token)

$ curl -v -H "Authorization: Bearer ${JWT}" -X PATCH --data-urlencode full_description@README.md https://hub.docker.com/v2/repositories/zenika/ztraining2strigo/
*   Trying 2600:1f18:2148:bc02:ab79:9619:af3f:8e4b:443...
* Connected to hub.docker.com (2600:1f18:2148:bc02:ab79:9619:af3f:8e4b) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.docker.com
*  start date: Jun 12 00:00:00 2022 GMT
*  expire date: Jul 11 23:59:59 2023 GMT
*  subjectAltName: host "hub.docker.com" matched cert's "*.docker.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> PATCH /v2/repositories/zenika/ztraining2strigo/ HTTP/1.1
> Host: hub.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Authorization: Bearer ...
> Content-Length: 10307
> Content-Type: application/x-www-form-urlencoded
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 415 Unsupported Media Type
< date: Wed, 13 Jul 2022 14:04:11 GMT
< content-type: application/json
< content-length: 42
< x-ratelimit-limit: 600
< x-ratelimit-reset: 1657721111
< x-ratelimit-remaining: 600
< x-trace-id: 09ccff23d18e55e9301e22d6981adf39
< server: nginx
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000
< 
{"message":"415: Unsupported Media Type"}
* Connection #0 to host hub.docker.com left intact

Looking at documentation, looks like x-www-form-urlencoded is not supported (anymore?).

$ curl -v -H "Authorization: Bearer ${JWT}" -H "Content-type: application/json" -X PATCH --data '{"full_description": "README.md"}' https://hub.docker.com/v2/repositories/zenika/ztraining2strigo/
*   Trying 2600:1f18:2148:bc01:33c8:2bac:60fa:e905:443...
* Connected to hub.docker.com (2600:1f18:2148:bc01:33c8:2bac:60fa:e905) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.docker.com
*  start date: Jun 12 00:00:00 2022 GMT
*  expire date: Jul 11 23:59:59 2023 GMT
*  subjectAltName: host "hub.docker.com" matched cert's "*.docker.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> PATCH /v2/repositories/zenika/ztraining2strigo/ HTTP/1.1
> Host: hub.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Authorization: Bearer ...
> Content-type: application/json
> Content-Length: 33
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Wed, 13 Jul 2022 14:13:58 GMT
< content-type: application/json
< content-length: 512
< x-ratelimit-limit: 600
< x-ratelimit-reset: 1657721698
< x-ratelimit-remaining: 600
< x-trace-id: 738f0a10a317d746025b6cb28d0b7426
< server: nginx
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000
< 
{"user":"zenika","name":"ztraining2strigo","namespace":"zenika","repository_type":"image","status":1,"description":"Create Strigo class for Zenika training.","is_private":false,"is_automated":false,"can_edit":false,"star_count":0,"pull_count":2277,"last_updated":"2022-07-13T13:56:40.069147Z","date_registered":"2021-03-03T16:01:09.749645Z","collaborator_count":0,"affiliation":null,"hub_user":"zigarn","has_starred":false,"full_description":"README.md","permissions":{"read":false,"write":false,"admin":false}}
* Connection #0 to host hub.docker.com left intact

=> Works :rocket:

[zappy-shu]

@zhangguanzhang unfortunately for now you would have to use a password. I will prioritize getting the DELETE tag flow migrated over to use the new authentication flows and support PATs ASAP. Sorry for the inconvenience

[zappy-shu]

@zigarn that's correct. URL data encoding is no longer supported. If this is a feature you want please could you open a new ticket? This one is getting a little full. Thanks

[zigarn]

@zappy-shu It's OK for me, just need to update https://circleci.com/developer/orbs/orb/circleci/docker#commands-update-description

[christian-korneck]

@zappy-shu thanks 🙇 a lot also for this also from my side! Updating the repo settings now seems to work fine. 🚀 (At a first look my "Docker Push Readme" Tool now seems to work with PATs without any change necessary).

Btw. I didn't need to create an admin scope PAT. Creating a read/write PAT from the Docker Hub WebUI seems to be sufficient.

[reitzig]

Extending on what @zigarn posted (thank you!), I needed to upload Markdown instead of a file name. I used to use this:

curl -siX PATCH "https://hub.docker.com/v2/repositories/${dh_repo}/" \
     -H "Authorization: JWT ${dh_token}" \
     -o "${CURL_OUT}" \
     --data-urlencode description="${gh_description}" \
     --data-urlencode full_description@"${readme_profile_filepath}"

which does not work anymore. Now I need to encode the file content as JSON:

curl -siX PATCH "https://hub.docker.com/v2/repositories/${dh_repo}/" \
     -H "Authorization: JWT ${dh_token}" \
     -H "Content-Type: application/json" \
     -o "${CURL_OUT}" \
     --data @- << PAYLOAD
{
    "description": "${gh_description}",
    "full_description": $(jq -Rsa . "${readme_profile_filepath}")
}
PAYLOAD

Seems to work!

Credits for the usage of jq: SO#50380697