Add ability to configure CredHub CLI HTTP timeout

[poblin-orange]

What version of the credhub server you are using?

2.5.6

What version of the credhub cli you are using?

2.5.3

If you were attempting to accomplish a task, what was it you were attempting to do?

Access credhub with cli

Get https://credhub.internal.paas:8844/api/v1/certificates: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

What did you expect to happen?

id expect to be able to define a larger timeout

What was the actual behavior?

Please confirm where necessary:

• [ ] I have included a log output
• [ ] My log includes an error message
• [ ] I have included steps for reproduction

If you are a PCF customer with an Operation Manager (PCF Ops Manager) please direct your questions to support (https://support.pivotal.io/)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/171391486

The labels on this github issue will be updated when the story is started.

[ankeesler]

@poblin-orange oh boy, this looks like another instance of the server taking forever to return data. Do you have a very large amount of certificates in your CredHub? We have seen this come up previously when CredHub had on the order of 100,000 certificates.

[poblin-orange]

@ankeesler not really. Hundreds of certs.

[ankeesler]

@poblin-orange - a couple more debug questions.

• If you hit the /api/v1/certificates endpoint with a raw curl command, do you still get the timeout? I want to validate my assumption that this is a timeout happening on the server side (and not the CLI).
• If you hit the /api/v1/certificates?name=/some-certificate endpoint, then do you still get a timeout? I want to see if we can narrow this issue down to a certain code path.
• Is the CLI timing out after about 45 seconds? I want to verify that it is this Timeout setting that we are hitting: https://github.com/cloudfoundry-incubator/credhub-cli/blob/e6410a077bac197ca006a8a2ee24f63bdbc824b7/credhub/client.go#L32.

If you need an immediate workaround, you can always rebuild the CLI with a much larger HTTP timeout: https://github.com/cloudfoundry-incubator/credhub-cli/blob/e6410a077bac197ca006a8a2ee24f63bdbc824b7/credhub/client.go#L32.

[poblin-orange]

@ankeesler in fact its slow server side for particular REST API (https://credhub-api.cfapps.io/version/2.5/#_get_all_certificates). Others are ok.

After double check, with have some secrets with a huge nomber of versions (2k in particular cases).

Ok for credhub cli rebuild. My initial request was about having an env var to set the timeout

[ankeesler]

@poblin-orange - thank you for looking into this and giving us this feedback. That is really helpful to know that the timeout is server side. We can try to repro on our side. We will also consider adding an env var to configure this timeout, as you suggested. Have you seen this env var pattern in other CLI's?

[poblin-orange]

@ankeesler we didnt meet timeouts for other CLIs. Seems that cf cli has a setting CF_DIAL_TIMEOUT (not the same, thats for initial cli access i guess).

[ankeesler]

@poblin-orange - do you have access to your CredHub logs for when this timeout happens? I just tried to fill my CredHub 2.5.5 with a ton of certificates (on the order of what you have) and couldn't reproduce.

While you are at it - what database are you using (e.g., local MySQL, local Postgres, remote MySQL, remote Postgres)? And what sort of IaaS disk type are you using (e.g., spinny disk, SSD)?

[ankeesler]

@poblin-orange we have added this feature in https://github.com/cloudfoundry-incubator/credhub-cli/commit/4ba6e6ae8c9d7de0ee7801ba11682393ad748b23. Can you please try it out and let us know if it meets your needs? Thank you.

[poblin-orange]

@ankeesler sorry missed ur previous message. (we are using a remote postgres, with iaas ssd disk). We will give it a try Thank you !