search

hashicorp / vault-service-broker

The official HashiCorp Vault broker integration to the Open Service Broker API. This service broker provides support for secure secret storage and encryption-as-a-service to HashiCorp Vault.
https://www.vaultproject.io/
Mozilla Public License 2.0
84 stars 38 forks source link

Expired token prevents service unbind #6

Closed Freakin closed 7 years ago

Freakin commented 7 years ago

I'm not terribly familiar with Vault yet so please forgive if I'm missing something

I created vault service broker and it is working correctly. I'm using the sample policy in the readme doc for the broker token.

I created a service successfully and bound it to my app, and was able to write secrets to the generic space backend and read them from my app (python app using hvac).

It appears my token has expired and I can no longer auth with it.

When I attempt to unbind my service, I get the following

cf unbind-service appname vaultservice
Unbinding app appname from service vaultservice in org org / space space as user@domain.com...
Unexpected Response
Response code: 502
CC code:       10001
CC error code: CF-ServiceBrokerBadResponse
Request ID:    63df3eda-87f3-4e17-7369-ddf1e030d506
Request ID:    0c1b0c37-6b67-4c66-6df2-f44ded7d4eaf::f55eefdf-895c-4e3e-8e15-f84f3ab88eee
Description:   Service instance vault: Service broker error: failed to revoke accessor 12312b67-3693-ac35-eb52-1c515cac547b: Error making API request.

URL: POST https://vault.domain.com:8200/v1/auth/token/revoke-accessor
Code: 400. Errors:

* 1 error(s) occurred:

* invalid accessor
FAILED

Here are logs from vault broker

2017-05-17T12:57:42.97-0700 [APP/0] OUT [INFO] unbinding service df0fe535-1143-4d24-9194-655bd2eee8ea for instance b8759394-5382-4c6c-8a00-4736d75cff18
2017-05-17T12:57:42.97-0700 [APP/0] OUT [DEBUG] reading cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.00-0700 [APP/0] OUT [DEBUG] decoding binding info for cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.00-0700 [APP/0] OUT [DEBUG] revoking accessor 29912b67-3693-ac35-eb52-1c515cac547b for path cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.01-0700 [RTR/4] OUT vault-broker.apps.domain.com - [17/05/2017:19:57:42.973 +0000] "DELETE /v2/service_instances/b8759394-5382-4c6c-8a00-4736d75cff18/service_bindings/df0fe535-1143-4d24-9194-655bd2eee8ea?plan_id=0654695e-0760-a1d4-1cad-5dd87b75ed99.shared&service_id=0654695e-0760-a1d4-1cad-5dd87b75ed99 HTTP/1.1" 500 0 260 "-" "HTTPClient/1.0 (2.7.1, ruby 2.2.4 (2015-12-16))" [REDACTED]:55945 x_forwarded_for:"[REDACTED], [REDACTED]" x_forwarded_proto:"https" vcap_request_id:fffdb4f3-8487-4ec3-7413-334e9b87a726 response_time:0.041498743 app_id:564d39aa-a080-4b28-a5f3-f862109ab73e
2017-05-17T12:57:43.01-0700 [RTR/4] OUT
2017-05-17T12:57:43.16-0700 [APP/0] OUT [INFO] unbinding service df0fe535-1143-4d24-9194-655bd2eee8ea for instance b8759394-5382-4c6c-8a00-4736d75cff18
2017-05-17T12:57:43.16-0700 [APP/0] OUT [DEBUG] reading cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.17-0700 [APP/0] OUT [DEBUG] decoding binding info for cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.18-0700 [APP/0] OUT [DEBUG] revoking accessor 29912b67-3693-ac35-eb52-1c515cac547b for path cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.18-0700 [RTR/3] OUT vault-broker.apps.domain.com - [17/05/2017:19:57:43.160 +0000] "DELETE /v2/service_instances/b8759394-5382-4c6c-8a00-4736d75cff18/service_bindings/df0fe535-1143-4d24-9194-655bd2eee8ea?plan_id=0654695e-0760-a1d4-1cad-5dd87b75ed99.shared&service_id=0654695e-0760-a1d4-1cad-5dd87b75ed99 HTTP/1.1" 500 0 260 "-" "HTTPClient/1.0 (2.7.1, ruby 2.2.4 (2015-12-16))" [REDACTED]:53419 x_forwarded_for:"[REDACTED], [REDACTED]" x_forwarded_proto:"https" vcap_request_id:3486f6f0-eadd-4fe9-533a-f3614ae0dcd8 response_time:0.026972471 app_id:564d39aa-a080-4b28-a5f3-f862109ab73e
2017-05-17T12:57:43.18-0700 [RTR/3] OUT
2017-05-17T12:57:43.34-0700 [APP/0] OUT [INFO] unbinding service df0fe535-1143-4d24-9194-655bd2eee8ea for instance b8759394-5382-4c6c-8a00-4736d75cff18
2017-05-17T12:57:43.34-0700 [APP/0] OUT [DEBUG] reading cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.36-0700 [APP/0] OUT [DEBUG] decoding binding info for cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.36-0700 [APP/0] OUT [DEBUG] revoking accessor 29912b67-3693-ac35-eb52-1c515cac547b for path cf/broker/b8759394-5382-4c6c-8a00-4736d75cff18/df0fe535-1143-4d24-9194-655bd2eee8ea
2017-05-17T12:57:43.37-0700 [RTR/1] OUT vault-broker.domain.com - [17/05/2017:19:57:43.336 +0000] "DELETE /v2/service_instances/b8759394-5382-4c6c-8a00-4736d75cff18/service_bindings/df0fe535-1143-4d24-9194-655bd2eee8ea?plan_id=0654695e-0760-a1d4-1cad-5dd87b75ed99.shared&service_id=0654695e-0760-a1d4-1cad-5dd87b75ed99 HTTP/1.1" 500 0 260 "-" "HTTPClient/1.0 (2.7.1, ruby 2.2.4 (2015-12-16))" [REDACTED]:59200 x_forwarded_for:"[REDACTED], [REDACTED]" x_forwarded_proto:"https" vcap_request_id:75a78298-5d81-4215-6e93-f22298f9d440 response_time:0.03506014 app_id:564d39aa-a080-4b28-a5f3-f862109ab73e
sethvargo commented 7 years ago

Hi @Freakin

This is actually a known issue. Unfortunately we haven't been able to think of a good way to solve this, since the token is needed to write (or in this case delete) the binding info from Vault.

Freakin commented 7 years ago

thanks @sethvargo. If I renew my token before ttl expiration will that allow me to use it indefinitely and delete it if desired?

sethvargo commented 7 years ago

@Freakin if it's a periodic token, yes. You can read more about periodic token creation in the README.

alwaysastudent commented 7 years ago

@sethvargo Slightly unrelated but I feel it is relevant in terms of maintaining data sanity. Should shutting down the broker, translate in cleaning up of the created mounts and token inside the vault server ? I was thinking if you could capturing the SIGINT signal and doing a graceful clean up, it would help in maintaining some data sanity. I am not a GO programmer and I am getting to know vault, so please forgive me for any ignorance.

But coming back to the issue in these lines, can you explain how availability/restarts of the broker would impact with the bound services ? I faced an issue the other day. I had my dev set up inactive for like 4 days and I started them up the other day to see the tokens set on the env for my bindings to be expired. The broker kept trying to renew the same and was getting back a 400. Finally I had to manually remove the mounts in vault server, purge the services and then recreate them.