vault-lb-aws module is not public

[kmcquade]

https://github.com/hashicorp-modules/vault-aws/blob/f-refactor/main.tf#L130

@bensojona

Just pointing it out :) I'm currently using aws_elb so I'd like to see what you're doing with it.

Also - looks like your module call to vault-lb-aws includes items for specifying a load balancer certificate. SSL Termination at a load balancer for any kind of hosted secrets management tool is definitely not best practice, although I suppose it's fair to offer your customers the ability to terminate SSL at the LB instead of E2EE. Do you use conditionals in the vault-lb-aws module to provide that as an option rather than requiring it in order to use the module? (Edit: I see that there's a "${var.create}" in the call. Might be worth pointing out the security tradeoff in a comment)

[bensojona]

Yep the ‘use_lb_cert’ var is where you can optionally choose to use it.

To be honest the only reason I added the LB was to make it easy to see the UI, and in doing so figured that optionally providing the ability to have certs when making the request made sense.

Can you shed some more light on why it’s not best practices? Not disagreeing, just trying to better understand.

On Mon, Apr 30, 2018 at 00:10 Kinnaird McQuade notifications@github.com wrote:

https://github.com/hashicorp-modules/vault-aws/blob/f-refactor/main.tf#L130

@bensojona https://github.com/bensojona

Just pointing it out :) I'm currently using aws_elb so I'd like to see what you're doing with it.

Also - looks like your module call to vault-lb-aws includes items for specifying a load balancer certificate. SSL Termination at a load balancer for any kind of hosted secrets management tool is definitely not best practice, although I suppose it's fair to offer your customers the ability to terminate SSL at the LB instead of E2EE. Do you use conditionals in the vault-lb-aws module to provide that as an option rather than requiring it in order to use the module?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp-modules/vault-aws/issues/16, or mute the thread https://github.com/notifications/unsubscribe-auth/ACUWapHUjA_Qn9bgND06xSi4DcbX-Jhiks5ttcnlgaJpZM4Tr0In .

-- Jon

[kmcquade]

Glad to provide some input. None of the below is meant to be overly critical - just want to properly answer the question. :)

With SSL Termination, traffic is no longer encrypted beyond the load balancer. This means that any secrets inside the HTTP headers or payloads included in the request are unencrypted. Anyone who obtains access to the contents of those unencrypted HTTP Requests could view tokens or secrets that are a part of that request - whether those are obtained via MiTM attack, via HTTP Proxy logs, any type of log aggregator agent forwarding plaintext requests, etc.

In a typical web application, SSL termination might not be an issue depending on the context. For example, if SSL Termination is used to leverage more capabilities of a WAF (checking user-supplied input that wouldn't be possible if viewing a fully encrypted request), but all sensitive data is encrypted within an unencrypted HTTP request (think: SSN or password is encrypted, but the HTTP request itself is not), then SSL Termination is a non-concern. However, in this case, secrets would be transmitted via plaintext - which presents risks mentioned above and probably violates compliance requirements for many customers.

That being said, I think I understand why you used that setup - it's a lot prettier for customers to not see the red "certificate mismatch" error. But perhaps there's another way to do it that would provide a more "secure by default" example.

[bensojona]

Thanks for the detailed response, that’s super helpful and and answers my question of why whenever I turned on verify_incoming on Consul, Vault, or Nomad it wouldn’t work!

I had assumed that by enabling ssl on the LB that the LB would present that cert to the target and thus client verification could be turned on.

This is probably a dumb question, but is there a way to turn off SSL termination on ALBs? I’m curious the best way to go about putting an LB in front of products that require TLS client verification, it must be a common method to abstract the ephemeral IPs of a service as you roll servers over time.

On Tue, May 1, 2018 at 03:55 Kinnaird McQuade notifications@github.com wrote:

Glad to provide some input. None of the below is meant to be overly critical - just want to properly answer the question. :)

With SSL Termination, traffic is no longer encrypted beyond the load balancer. This means that any secrets inside the HTTP headers or payloads included in the request are unencrypted. Anyone who obtains access to the contents of those unencrypted HTTP Requests could view tokens or secrets that are a part of that request - whether those are obtained via MiTM attack, via HTTP Proxy logs, any type of log aggregator agent forwarding plaintext requests, etc.

In a typical web application, SSL termination might not be an issue depending on the context. For example, if SSL Termination is used to leverage more capabilities of a WAF (checking user-supplied input that wouldn't be possible if viewing a fully encrypted request), but all sensitive data is encrypted within an unencrypted HTTP request (think: SSN or password is encrypted, but the HTTP request itself is not), then SSL Termination is a non-concern. However, in this case, secrets would be transmitted via plaintext - which presents risks mentioned above and probably violates compliance requirements for many customers.

That being said, I think I understand why you used that setup - it's a lot prettier for customers to not see the red "certificate mismatch" error. But perhaps there's another way to do it that would provide a more "secure by default" example.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp-modules/vault-aws/issues/16#issuecomment-385442752, or mute the thread https://github.com/notifications/unsubscribe-auth/ACUWauwt_xkwHyM11HAGwHl8RHlHO4fRks5ttzQDgaJpZM4Tr0In .

-- Jon

[kmcquade]

verify_incoming

Yeah... verify_incoming gave me a little bit of hell when I started. I see why you thought that it would (AWS-signed Certs and all) but I'm guessing that they don't have extendedKeyUsage clientAuth enabled. Outdated but helpful article that references this. That is probably the source of your issue... the certificates provided by the ALB aren't for client authentication.

Way to turn off SSL Termination on ALBs

To answer your question directly: From what I remember, you have the option built in to your module. However, leveraging ALBs effectively pretty much requires SSL Termination (see tradeoffs here), which isn't suitable for Vault as a secrets management solution that many of your customers rely on for compliance.

To answer your intended question: You don't want to use ALBs for this - you want to use NLBs (Terraform groups this under one resource - aws_lb, with two different types). ALBs = Layer 7 (HTTP and HTTPS) NLBs = Layer 4 (TCP) Classic Load balancers: Kinda both. https://aws.amazon.com/elasticloadbalancing/details/#compare

Considering that Consul requires (depending on your use case + architecture) more than incoming port 443 - it's prudent to use a general NLB module that can be applied for accessing both Vault and Consul. That, and it's required for hosting VPC Endpoint Interfaces - a very cool use case for Vault, Consul, and pretty much any service that can get behind an NLB.

[bensojona]

So would we need a network load balancer for TCP traffic and a separate application load balancer for HTTP/HTTPS traffic from the UI?

On Thu, May 3, 2018 at 14:00 Kinnaird McQuade notifications@github.com wrote:

verify_incoming

Yeah... verify_incoming gave me a little bit of hell when I started. I see why you thought that it would (AWS-signed Certs and all) but I'm guessing that they don't have extendedKeyUsage clientAuth enabled. Outdated but helpful article that references this https://www.mauras.ch/securing-consul.html#tls. That is probably the source of your issue... the certificates provided by the ALB aren't for client authentication. Way to turn off SSL Termination on ALBs

To answer your question directly: From what I remember, you have the option built in to your module. However, leveraging ALBs effectively pretty much requires SSL Termination (see tradeoffs here https://stackoverflow.com/questions/42027582/application-load-balancer-elbv2-ssl-pass-through#answer-42027781), which isn't suitable for Vault as a secrets management solution that many of your customers rely on for compliance.

To answer your intended question: You don't want to use ALBs for this - you want to use NLBs (Terraform groups this under one resource - aws_lb, with two different types). ALBs = Layer 7 (HTTP and HTTPS) NLBs = Layer 4 (TCP) Classic Load balancers: Kinda both. https://aws.amazon.com/elasticloadbalancing/details/#compare

Considering that Consul requires (depending on your use case + architecture) more than incoming port 443 - it's prudent to use a general NLB module that can be applied for accessing both Vault and Consul. That, and it's required for hosting VPC Endpoint Interfaces - a very cool use case for Vault, Consul, and pretty much any service that can get behind an NLB.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp-modules/vault-aws/issues/16#issuecomment-386173281, or mute the thread https://github.com/notifications/unsubscribe-auth/ACUWat1zy7IhGf3QIccR3f9yjgZtHCB_ks5tumTZgaJpZM4Tr0In .

-- Jon

[kmcquade]

Nope. Pass the HTTPS traffic through the NLB to the Vault nodes. No ALB at all. If using Consul, just create multiple listeners.

[bensojona]

Ah interesting, I’ll give it a shot, thanks!

On Fri, May 4, 2018 at 08:06 Kinnaird McQuade notifications@github.com wrote:

Nope. Pass the HTTPS traffic through the NLB to the Vault nodes. No ALB at all. If using Consul, just create multiple listeners.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp-modules/vault-aws/issues/16#issuecomment-386419762, or mute the thread https://github.com/notifications/unsubscribe-auth/ACUWasuFLdJSeFyyfVepU20AXhOAuBjZks5tu2M9gaJpZM4Tr0In .

-- Jon