Admin user cannot rotate token

[apr-1985]

Artifactory version: 7.33.12 Vault Version: 1.12.2 Artifactory Plugin version: 0.2.0

I have set the Artifactory vault plugin following the instructions here https://www.jfrog.com/confluence/display/JFROG/Hashicorp+Vault+Artifactory+Secrets+Plugin

However when I try to rotate the admin token as documented I get an error

❯ vault write -f artifactory/config/rotate
Error writing data to artifactory/config/rotate: Error making API request.

URL: PUT https://vault.<redacted>/v1/artifactory/config/rotate
Code: 400. Errors:

* error revoking existing AccessToken

I cannot see any errors in the Artifactory logs and in-fact the logs looks like there is success in creating the new token

2022-12-19T10:23:23.066Z|418fe7ca2db009c7|UNKNOWN|UNKNOWN|admin|jfac@01e7hr4k76vr470nkn64m81n3n/users/admin|C|TKN|{"added":{"owner":"jfac@01e7hr4k76vr470nkn64m81n3n","created":"1671445403063","expirationTime":"1702981403063","subject":"jfac@01e7hr4k76vr470nkn64m81n3n/users/admin","scope":"applied-permissions/admin","id":"2662c852-b527-40a9-a6c3-8b328d6fde2d","type":"generic","username":"admin"}}

10:23:27.102
2022-12-19T10:23:23.038Z|4f01f51bfd85b52|<redacted>|token:admin|GET|/api/system/version|200|-1|0|13|art-secrets-plugin

10:23:27.326
2022-12-19T10:23:23.038Z [4f01f51bfd85b52 ] [ACCEPTED LOGIN]   for client : token:admin / <redacted>

But there doesn't seem to be anything in the logs for even trying to revoke the old token

[TJM]

This is interesting, according to the error:

* error revoking existing AccessToken

... the token was "rotated" successfully, but it had a problem revoking the initial admin token. Revoking the old token happens at the very end. https://github.com/jfrog/artifactory-secrets-plugin/blob/v0.2.0/path_config_rotate.go#L93

Was the new token working otherwise? Maybe we need to make that a less critical failure? (WARNING?)

[TJM]

I was unable to reproduce, but I am trying to run artifactory-jcr, as it doesn't require a license. Was your initial admin token setup with an expiration date by any chance? (actually even when I set an expiration date, it still works)...

Here is the test setup I am using:

First setup Artifactory (JCR)

• minikube start
• helm upgrade --install jcr jfrog/artifactory-jcr --version 107.33.12
• (wait a few minutes for all the images to download, start and run... until kubectl get pods shows three running pods)
• kubectl port-forward jcr-artifactory-0 8082
• Visit http://127.0.0.1:8082
  • NOTE: I did not get the "onboarding wizard" which means I did not get a chance to sign the EULA, but I was still able to get into the Access Token screen and generate a new token (default creds are admin/password)
  • Create a new Access Token scoped, admin, username admin (doesn't seem to matter), No Expiration (again, doesn't seem to matter)

Next Setup Vault and artifactory-secrets-plugin

(In another window)

• Download VAULT. I am using "asdf"...
  • asdf install vault 1.12.2
  • asdf local vault 1.12.2
• download or clone this artifactory-secrets-plugin code
  • git clone https://github.com/jfrog/artifactory-secrets-plugin.git
  • cd artifactory-secrets-plugin
• make start (in the cloned code directory for artifactory-secrets-plugin) which starts up a dev vault server with the plugin added

Lastly, Configure the plugin

(In yet another window)

• export VAULT_ADDR='http://127.0.0.1:8200'
• export VAULT_TOKEN=root
• ARTIFACTORY_TOKEN=(TOKEN FROM ARTIFACTORY ABOVE)
• vault secrets enable artifactory
• vault write artifactory/config/admin url=http://127.0.0.1:8082/artifactory access_token=$ARTIFACTORY_TOKEN
• vault write -f artifactory/config/rotate

[tmcneely@local ~]$ vault write artifactory/config/admin url=http://127.0.0.1:8082/artifactory access_token=$ARTIFACTORY_TOKEN
Success! Data written to: artifactory/config/admin
[tmcneely@local ~]$ vault write -f artifactory/config/rotate
Success! Data written to: artifactory/config/rotate

[apr-1985]

Thanks for looking at this.

It is quite an interesting one.... Using the CLI to generate a token without expiry I get a certificate error :/ Using a token from the UI it seems that even when selecting to not expire the tokens get an expiry time and the revocation fails.

Through the CLI

First generate an admin token with no expiry (as documented https://www.jfrog.com/confluence/display/JFROG/Hashicorp+Vault+Artifactory+Secrets+Plugin)

curl -XPOST -u admin:AK<SNIP>e "https://my.artifactory.systems/artifactory/api/security/token" \
    -dusername=admin \
    -dexpires_in=0 \
    "-dscope=member-of-groups:*"
{
  "access_token" : "ey<SNIP>vqgQ",
  "expires_in" : 0,
  "scope" : "member-of-groups:*",
  "token_type" : "Bearer"
}%

Enable a new AF secrets engine

vault secrets enable -path af_new artifactory
Success! Enabled the artifactory secrets engine at: af_new/

Write the config with the token from above

vault write af_new/config/admin url=https://my.artifactory.systems/artifactory access_token="ey<SNIP>vqgQ"
Success! Data written to: af_new/config/admin

Try to rotate

vault write -f af_new/config/rotate
Error writing data to af_new/config/rotate: Error making API request.

URL: PUT https://vault.dev.systems/v1/af_new/config/rotate
Code: 400. Errors:

* error parsing existing AccessToken: could not get the certificate: HTTP response 401

Vault logs

2023-01-06T10:27:13.249Z [WARN]  secrets.artifactory.artifactory_505c9f17.artifactory.artifactory: got non-200 status code: statusCode=401 timestamp=2023-01-06T10:27:13.248Z
2023-01-06T10:27:13.249Z [ERROR] secrets.artifactory.artifactory_505c9f17.artifactory.artifactory: error retrieving root cert: err="could not get the certificate: HTTP response 401" timestamp=2023-01-06T10:27:13.248Z

I cannot really work this error out. There isn't anything to go on. The token is valid as is the address and can be used for a standard curl command.

curl -uadmin:ey<SNIP>vqgQ https://my.artifactory.systems/artifactory/api/repositories
[ {
  "key" : "Generic-Core-Packages-Local",
  "description" : "Core Packages test",
  "type" : "LOCAL",
<SNIP>

Using a Token generated from the UI Even when selecting Never Expire the token looks to have an expiry of 31536000 Seconds according to the screen where the token is visible

vault write af_new/config/admin url=https://my.artifactory.systems/artifactory access_token="ey<SNIP>3npzA"
Success! Data written to: af_new/config/admin

Then try to rotate

vault write -f af_new/config/rotate
Error writing data to af_new/config/rotate: Error making API request.

URL: PUT https://vault.dev.systems/v1/af_new/config/rotate
Code: 400. Errors:

* error revoking existing AccessToken

Vault logs

2023-01-06T10:30:20.434Z [WARN]  secrets.artifactory.artifactory_505c9f17.artifactory.artifactory: got non-200 status code: statusCode=400 timestamp=2023-01-06T10:30:20.434Z

When doing this rotation it seems that a new token is generated in Artifactory also with an expiry

admin
e586815f-9e97-<SNIP>-e6e42446a126
06-01-23 10:31:27 +0000
06-01-24 10:31:27 +0000

I am not sure which token Vault is using though.

I am using Artifactory Enterprise in a three node cluster version 7.33.12 running on EC2 behind an ALB with and ACM certificate. Unfortunately I am not in a position to upgrade this cluster at the moment to test the latest version of Artifactory but there does seem to be some issue in the token handling (the UI error at least :) ).

Thanks again for looking at this.

[apr-1985]

When getting a token through the UI, it shows up in the show non-revocable tokens list which I guess explains why vault cant revoke it. TBH I am not sure why there are non-revocable tokens in Artifactory at all as that is clearly a security issue but that is an aside I guess.

So I guess my issues in the bug currently stand at:

• why am I getting a certificate error when using a token with no expiry token generated from the command line
• why do access tokens in the UI generated with no expiry still get an expiry and become non-revocable

[TJM]

@apr-1985 - The documentation should be re-worded. When it says:

In Artifactory 7.4+ this can be done in the UI ...

... what it should probably say is:

In Artifactory 7.4+, you must generate an access token in the UI ....

The /artifactory/api/security/token path is deprecated, and apparently generates a different type of token than /access/ API.

• https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-CreateToken(Deprecated)

I think in some more recent version, they have re-added API level access to create admin tokens from passwords, but we used a shell script to "emulate" a browser using curl with the --cookie-jar option in our automation.

Can you please test using a token from the UI and see if it works as desired for rotation? (you might save your old token and see if it still works after revoking it too as per #22 )

[apr-1985]

When I use a token from the UI I get the error * error revoking existing AccessToken see the explanation in the comment https://github.com/jfrog/artifactory-secrets-plugin/issues/23#issuecomment-1373463435

I think the issue is that UI tokens (even when no expiry set) and (and then ones created by Vault) are being created as non-revocable tokens with a 1 year expiry which is why I am seeing these issues.

[TJM]

The token generated from CLI at /artifactory/api is not valid for /access/api.

The plugin detects the version of artifactory, and if it is newer than 7.12.0, it tries to retrieve the root certificate that signs the JWT Tokens from/access/api/v1/cert/root ... As you noted, with that CLI token, you get a 401.

[tmcneely@local ~]$ curl -u admin:password "http://127.0.0.1:8082/artifactory/api/security/token" \
    -dusername=admin \
    -dexpires_in=0 \
    "-dscope=member-of-groups:*"
{
  "access_token" : "eyJ(SNIP)dA",
  "expires_in" : 0,
  "scope" : "member-of-groups:*",
  "token_type" : "Bearer"
}%
[tmcneely@local ~]$ token=eyJ(SNIP)dA"
[tmcneely@local ~]$ curl -H "Authorization: Bearer $token" http://127.0.0.1:8082/access/api/v1/cert/root
{
  "errors" : [ {
    "code" : "UNAUTHORIZED",
    "message" : "Bearer authentication failed, invalid token"
  } ]
}

For what it's worth, you do not need to rotate the token, it is just a best practice. However, I doubt that the /artifactory/api token would work later to create a "user" token either, since the plugin would again try to use the /access/api instead of the deprecated "/artifactory/api" endpoints. If you want (or have) to use command line created tokens, I have pasted a couple of our scripts (thats a link)... you can snag the curl commands out of them, but basically it logs in like the WebUI, saves the cookies in a file named "cookiejar" then passes that back to the UI command to generate a token. This code works on our 7.30+ servers, but no warranty, use at your own risk, of course :)

Also, I created the initial admin token in the UI, with a 1 day expiration, and it still worked fine when rotated, so I think it is still something else.

[alexhung]

I don't have any answer for you guys yet but note that there's enhancement to token generation without using the JFrog Platform UI or via another token in 7.38.4

[alexhung]

@apr-1985 - The documentation should be re-worded. When it says:

In Artifactory 7.4+ this can be done in the UI ...

... what it should probably say is:

In Artifactory 7.4+, you must generate an access token in the UI ....

The /artifactory/api/security/token path is deprecated, and apparently generates a different type of token than /access/ API.

• https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-CreateToken(Deprecated)

I think in some more recent version, they have re-added API level access to create admin tokens from passwords, but we used a shell script to "emulate" a browser using curl with the --cookie-jar option in our automation.

Can you please test using a token from the UI and see if it works as desired for rotation? (you might save your old token and see if it still works after revoking it too as per #22 )

I'm in the process of updating the README as part of investigating a bug. The incorrect API path is part of this update.

[alexhung]

@mritunjaykumar ☝🏼

[TJM]

I don't have any answer for you guys yet but note that there's enhancement to token generation without using the JFrog Platform UI or via another token in 7.38.4

Thanks @alexhung ... I knew that had been created, as we were one of the requestors, but by then we had already written the scripts linked above to handle that. I suppose at some point we will have to try to use that functionality, but kubernetes makes it more interesting. I suppose we could kubectl exec or kubectl cp it? ;) I will try to stay on topic here, suffice it to say that the only "supported" mechanism to generate an initial admin scoped access token is the UI, for any reasonably up to date Artifactory. :) (I mean 7.4.x was a long time ago since they are on like 7.49.x now) - https://www.jfrog.com/confluence/display/JFROG/Artifactory+End+of+Life

[apr-1985]

Thanks I will definitely have a play on Monday and see if I can get some sort of token working as I really want this to work 😄

I am really quite worried about the access tokens implementation in the Artifactory version I have. It seems rather odd and quite buggy (not respecting never expire in the ui for example), and having tokens that cannot be revoked is just a no go from a security standpoint. Do you know the reasoning for why non-revocable tokens exist?

Vault not being able to revoke tokens at the end of the lease means that the provider is not compatible with the AF version I am using in my opinion as this is the fundamental use case.

I will try and spin up an instance of a newer Artifactory version and test but I don't have any spare licences at the moment (hence why I haven't upgraded) as we are in renewal negotiations so might be several weeks till I can free one up.

[TJM]

FWIW, we are an enterprise+ customer, but I have been using artifactory-jcr (JFrog Container Registry) to do testing without constantly requesting trial licenses (which causes a lot of spam from the sales droids) :)

[alexhung]

I don't have any answer for you guys yet but note that there's enhancement to token generation without using the JFrog Platform UI or via another token in 7.38.4

Thanks @alexhung ... I knew that had been created, as we were one of the requestors, but by then we had already written the scripts linked above to handle that. I suppose at some point we will have to try to use that functionality, but kubernetes makes it more interesting. I suppose we could kubectl exec or kubectl cp it? ;) I will try to stay on topic here, suffice it to say that the only "supported" mechanism to generate an initial admin scoped access token is the UI, for any reasonably up to date Artifactory. :) (I mean 7.4.x was a long time ago since they are on like 7.49.x now) - https://www.jfrog.com/confluence/display/JFROG/Artifactory+End+of+Life

The inability to create the initial admin scoped token without going through UI (or use private UI API) is a known issue. I don't have any more information to provide, unfortunately.

[alexhung]

@apr-1985 While I don't have answers for your issue right now, I can tell you that lots of effort has been expended last year by the JFrog Access team to improve the access token support.

I recommend the following:

1. Spin up a test version using latest Artifactory version
2. Test out your use cases/scenarios
3. Report any issue/bug/missing feature to customer support, with as concise information as possible. Ideally with reproducible steps.

[apr-1985]

@apr-1985 While I don't have answers for your issue right now, I can tell you that lots of effort has been expended last year by the JFrog Access team to improve the access token support.

I recommend the following:

1. Spin up a test version using latest Artifactory version

2. Test out your use cases/scenarios

3. Report any issue/bug/missing feature to customer support, with as concise information as possible. Ideally with reproducible steps.

Thanks @alexhung. I totally agree with your point which I why i clarified with the version I am using as I cannot speak for the newer versions 😀

If there is any further help I can provide until I get new licenses please do let me know.

[TJM]

Here is an adaptation of our script using the undocumented (private) UI APIs to get an access token without manually logging in... https://gist.github.com/TJM/028ca421068d518f943685dffc828df4#file-getartifactoryadmintoken-sh

I am also testing with the same version you specified 7.33.12, which makes me wonder if you started with a "fresh" install in a lab or whatever, if you would run into the same thing, or if there is some setting either in a yaml/xml or maybe database which would be affecting access tokens/revoking?

[alexhung]

We also have a simpler one for development use in the Terraform Provider repo: https://github.com/jfrog/terraform-provider-artifactory/blob/master/scripts/get-access-key.sh

[apr-1985]

Just seen on the release notes that there is a bug fixed

Fixed an issue whereby, access tokens created in Artifactory versions prior to 7.39.0, could not be revoked

https://www.jfrog.com/confluence/display/JFROG/Artifactory+Release+Notes#ArtifactoryReleaseNotes-Artifactory7.50.7Cloud

As I am using 7.33.12 this is most likely the cause of the issue I am seeing and why it is working for everyone else.

I am still a couple of weeks away from upgrading my clusters (enterprise politics) but I think this issue can be closed as fixed in a later version if you all agree.

[alexhung]

@apr-1985 Do you have any update on your end? Let us know if you've upgraded your Artifactory and this is still an issue.

[alexhung]

Closing due to inactivity. Feel free to re-open with more information, logs, etc.