Authorization error when connection to Vault through HAProxy

[xskrasek]

Describe the bug I have vault hosted behind HAProxy with TLS Bridging, as termination is not possible for us. However, when I access vault through the HAProxy with root token, I do not have permission to e.g. read some secret, where as when I connect to a node directly, it works as expected. When I try this in CLI, I get "tls: failed to verify certificate: x509: certificate signed by unknown authority" even though browser does not complain and when I use -tls-skip-verifyI get

Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path

Expected behavior Root user can work with Vault when connecting through HAProxy.

[maxb]

Based on what you've said so far, this sounds like a problem with your deployment, rather than a bug in Vault.

https://discuss.hashicorp.com/ is generally considered a more appropriate venue for things that need to be fixed by you in your environment, reserving the issue tracker for things that need to be fixed in the Vault project.

Still, I'll try to provide some starting points for investigation:

I have vault hosted behind HAProxy with TLS Bridging, as termination is not possible for us.

Is "TLS Bridging" a standard term? I had to Google it, and it seems that it probably means that you are terminating the TLS at haproxy, but then you're having haproxy establish an entirely separate TLS connection to deliver the request to the backend.

However, when I access vault through the HAProxy with root token, I do not have permission to e.g. read some secret, where as when I connect to a node directly, it works as expected.

Root tokens, by definition, have permission to do anything, so it cannot be possible that you "do not have permission" - something else must be going wrong.

When I try this in CLI, I get "tls: failed to verify certificate: x509: certificate signed by unknown authority" even though browser does not complain

Presumably this means your browser has been configured to trust your internal CA, but your internal CA has not been added to the certificate store used by the Vault CLI (which just uses Go's default behaviour).

when I use -tls-skip-verifyI get

Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path

OK, so once we put TLS CAs to one side, you get an HTTP 403. The "preflight capability check" is the request the Vault CLI's vault kv ... commands make implicitly, to the sys/internal/ui/mounts/... endpoint to check whether they are dealing with a v1 or v2 KV.

The fact you are getting a 403 makes me wonder if your Vault token isn't actually being passed through haproxy to the Vault server at all, leading to your request being processed as an anonymous request.

[xskrasek]

Thank you for your reply :)

Is "TLS Bridging" a standard term? I had to Google it, and it seems that it probably means that you are terminating the TLS at haproxy, but then you're having haproxy establish an entirely separate TLS connection to deliver the request to the backend.

Yes. In Haproxy configuration language, termination means ending tls at proxy and then communicating in plaintext, that is what I meant.

However, when I access vault through the HAProxy with root token, I do not have permission to e.g. read some secret, where as when I connect to a node directly, it works as expected.

Root tokens, by definition, have permission to do anything, so it cannot be possible that you "do not have permission" - something else must be going wrong.

I know, but somehow I get an error when using the token, which is why I reached out here.

When I try this in CLI, I get "tls: failed to verify certificate: x509: certificate signed by unknown authority" even though browser does not complain

Presumably this means your browser has been configured to trust your internal CA, but your internal CA has not been added to the certificate store used by the Vault CLI (which just uses Go's default behaviour).

when I use -tls-skip-verifyI get

Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path

OK, so once we put TLS CAs to one side, you get an HTTP 403. The "preflight capability check" is the request the Vault CLI's vault kv ... commands make implicitly, to the sys/internal/ui/mounts/... endpoint to check whether they are dealing with a v1 or v2 KV.

The fact you are getting a 403 makes me wonder if your Vault token isn't actually being passed through haproxy to the Vault server at all, leading to your request being processed as an anonymous request.

I am able to browse the secret engines and kv structure, but I can not see any of the actual secrets. If the proxy did not pass the token at all, I think I should not be able to get the paths inside Vault, this part is confusing to me.

[maxb]

It's probably best to ignore the browser for now, because browsers are complicated, whereas with the CLI it is easy to describe exactly what you do, and what the response is.

I suggest using

vault read auth/token/lookup-self

to have Vault confirm to you what token it is seeing, once the request gets to it.

[xskrasek]

Turns out, that the preflight check failed because of error on my side, sorry for that.

I tried to lookup token and it seems to be correct. It returns the same value as when talking to the node directly and I am also able to retrieve the secrets. Therefore the issue seems to only be with the browser, where I can browse but not retrieve the actual secret values. I get this error.

Your auth token does not have access to sys/capabilities-self. Vault Enterprise uses this endpoint to determine what actions are allowed in the interface.

Make sure the policy for the path sys/capabilities-self includes capabilities = ['update'].

Honestly, I am not able to tell, whether this is caused by my configuration or if this is some bug in Vault frontend. I encountered some issues with that before, so it is in my opinion also an option.

[xskrasek]

I was able to track this down. I inspected traffic in the browser and got Origin not allowed. I found out that my CORS settings are different than I thought. After fixing those, it now works correctly. However, I think that the error which was shown by the vault frontend is not correct, as it did not describe the issue and was misleading.

[maxb]

So, to summarize, this is actually a bug report about the Vault web UI returning misleading errors, when a loadbalancer mangles the CORS headers?

[xskrasek]

Yes, thank you for the summary :) . To make sure, the issue is that the load balancer has its own address, which was not allowed in CORS and therefore, when accessing vault using the load balancer's address, the secret values did not load. However, Vault shows that the token does not have capabilities. After allowing the load balancer address in CORS, it now functions correctly.

[heatherezell]

So, to summarize, this is actually a bug report about the Vault web UI returning misleading errors, when a loadbalancer mangles the CORS headers?

Thank you, Max! I'll direct this to our UI team as a result.

[zofskeez]

@xskrasek do you have any suggestions on how I can recreate this locally without a load balancer? I'm looking at surfacing the raw error message but I would like to test this exact scenario to make sure the relevant information is being displayed in the UI.