Plugin Vault: status 307 Temporary Redirect

stackshadow commented 1 year ago

Relevant telegraf.conf

[[inputs.vault]]
      url = "http://vaulty-0.dev:8200"
      token = "XXX"
      interval = "30s"

      [inputs.vault.tags]
        namespace = "dev"

Logs from Telegraf

2023-10-18T09:23:30Z E! [inputs.vault] Error in plugin: http://vaulty-0.dev:8200/v1/sys/metrics returned HTTP status 307 Temporary Redirect

System info

Telegraf 1.28.2 on Kubernetes v1.26.3

Docker

No response

Steps to reproduce

Use Vault in a cluster with 3 nodes
Add config to scrape all 3 nodes
Scrape of "standby"-node emit the "redirect-error" in the logs

Expected behavior

Follow the redirect

Actual behavior

Only the master-node will be scraped and for standby nodes we got these redirect-error

Additional info

No response

powersj commented 1 year ago

Hi,

Thanks for the report. Is this a new issue that was previously working? Or has this always occurred?

Follow the redirect

Do you happen to know what the redirected URLs are? If we do follow the redirect, does the new page actually produce real data?

Thanks

stackshadow commented 1 year ago

Well i think thats not an regression. So its not working before :)

Vault itself is in cluster mode, so one of the 3 nodes is the "leader" and all other vault-nodes are in standby. When an API-Call is made to an standby node, an 307 ( redirect ) will be done to the "leader" node. An curl will follow the http-redirect, but somehow this plugin does it not :)

For example:

We have three nodes:

an curl will return this when do on a standby node:

curl -v http://vaulty-0.dev:8200/v1/sys/metrics returns:

> GET /v1/sys/metrics HTTP/1.1
> Host: vaulty-0.dev:8200
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 307 Temporary Redirect
< Cache-Control: no-store
< Location: http://vaulty-1.dev:8200/v1/sys/metrics
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Date: Thu, 19 Oct 2023 06:43:48 GMT
< Content-Length: 0
< 
* Connection #0 to host vaulty-0.dev left intact

so http://vaulty-0.dev:8200 will be redirected to http://vaulty-1.dev:8200

and when the -L is used in curl, it will follow the redirect-url

https://everything.curl.dev/http/redirects#tell-curl-to-follow-redirects

I've build some internal telegraf plugins myself, so maybe i have on the weekend spare time to debugging this 😄

srebhan commented 1 year ago

@stackshadow can you please test the binary in #14153 available once CI finished the tests successfully. Let me know if this fixes the issue!

stackshadow commented 12 months ago

I had some wild week, thanks for your support, i will check if this works 👍

influxdata / telegraf