400 Grant Token has been revoked, has expired, or the refresh token is invalid

[DylanWoodrow]

Writing an issue report on behalf of a Venafi customer making use of this tool.

Customer is experiencing a 400 Grant Token has been revoked, has expired, or the refresh token is invalid error when attempting step 10 of the readme. Token is valid from the Venafi side, and can successfully authenticate outside of this tool. Resetting the tool and walking back through setup tasks still has not allowed the customer to successfully proceed.

Customer is using version 20.3 of TPP. Let me know what other information I can provide to aid diagnosis and troubleshooting.

[tr1ck3r]

@DylanWoodrow can you please provide some detail on what "can successfully authenticate outside of this tool" means? You can use the access token to request a certificate using VCert CLI enroll action? You can use the access token to make WebSDK calls? (if so which API methods did you invoke?)

Also, can you please describe what the Vault architecture looks like? Cluster or standalone? If a cluster, how many Vault servers and what, if any, types of replication are being used.

Lastly, how was the token requested? Using VCert CLI or direct POST /vedauth/authorize REST API call? Either way what values were used for the client_id and scope?

[DylanWoodrow]

Reply from the customer:

"I can request the certs from Venafi via both Postman and my local Vault instance. There doesn’t seem be any issue at Venafi end.

We are using Vault cluster with 5 servers Raft-Integrated-Storage which replicate the data."

[tr1ck3r]

Thanks @DylanWoodrow. I'm assuming they're seeing the 400 Grant Token has been revoked, has expired, or the refresh token is invalid error in the Vault log, right? Is that actually preventing the vault-pki-monitor-venafi from working (enforcing policy and/or pushing issued certificates into TPP)? The way this solution was designed it immediately uses the refresh token (if provided) to obtain a new pair of tokens. If there are multiple Vault servers (i.e. a cluster) then obviously only one will be able to successfully perform that task and the others would get the 400 error... but once that new access token is obtained and the venafi secret is updated, all the Vault servers should be able to use it. That's a long winded way of saying it is expected behavior to see those errors in the Vault log when our solution is in the process of obtaining a new pair of tokens.

[DylanWoodrow]

They are getting the error on the command line when attempting to invoke:

$ vault write pki/venafi/tpp \
    url="https://tpp.example.com" trust_bundle_file="/path/to/bundle.pem" \
    access_token="<token>==" refresh_token="<refresh token>=="

Per step 10 of the readme - it is erroring out trying to accept the initial attempt to write the trust

[tr1ck3r]

Thanks for the clarification. So that's the expected behavior. The first time that command is executed, the refresh token is used by our solution to obtain a new pair of tokens (security consideration ensuring that no one other than this venafi secret in Vault can use this grant). There's no need to execute that command again unless you want to change to a different grant.

[DylanWoodrow]

Customer reports this error when they attempt to continue on from step 10 despite the 400 grant token error:

Error writing data to venafi-pki-monitor/venafi-policy/default: Error making API request.
URL: PUT https://vault-enterprise-dev.corp.internal.citizensbank.com:8200/v1/venafi-pki-monitor/venafi-policy/default
Code: 500. Errors:
* 2 errors occurred:
* errors from both primary and secondary; primary error was expected Venafi secret but got nil from Vault storage <nil>; secondary errors follow
* expected Venafi secret but got nil from Vault storage <nil>

If the 400 grant error is expected behavior - what would be the best way to confirm that an attempt to write the access and refresh tokens was successful?

Prior to this, I confirmed with the customer that they have run steps 7-9 and those commands reported success.

[tr1ck3r]

@DylanWoodrow when you have a chance can you please send an excerpt of the Vault server log from approximately 5 minutes before and 5 minutes after executing the step 10 and 11 commands to opensource@venafi.com?

Also, can you confirm that after running the step 10 command and getting the 400 error the same access token from that command works without error using VCert CLI or direct WebSDK call? Trying to ascertain whether the error pertains to the tokens being passed in the command and to previous tokens that are stored in the venafi secret. If the access token still works after running that command then that confirms Vault did not (successfully) use the refresh token passed in the command.

[DylanWoodrow]

The logs requested were emailed to the above address. Afterwards, the following suggestion was given:

based on the Vault logs you sent it looks like the storage for the venafi secret may have gotten corrupted somehow. My suggestion would be for them to try to create a new venafi secret (e.g. different name like "tpp2" instead of "tpp") then reference that secret in the subsequent steps and see if that works.

Which was not successful on the customer's end.:

"Error writing data to <path>/tpp2: Error making API Request.

URL: Put <URL>/tpp2
Code: 400. Errors:
* errors from both primary and secondary; primary error was unexpected status code on TPP Authorize. Status: 400 Grant has been revoked, has expired, or the refresh token is invalid; secondary errors follow
* Unexpected status code on TPP Authorize. Status: 400 Grant has been revoked, has expired, or the refresh token is invalid.

[DylanWoodrow]

Update from the customer:

I removed and reinstated a vault appRole on the box and apparently that fixed the issue. I can now configure the plugin and issue the certs. I am don’t what the root cause is here. I’ll do a some more testing to verify everything is working as expected and update you guys on how it goes.

It looks like the original role may have been corrupting secrets, judging by the previous errors and troubleshooting we went through. Is there anything further that can and should be done to diagnose why this was happening to begin with?

[luispresuelVenafi]

Hi Dylan. I believe that from our side, anything left was to check the customer's environment like making sure his storage as alright, proper Vault settings, etc. For what it was mentioned, the issue seems to be related to an authentication issue but not the tool (the Venafi plugin) itself.