search

aws / aws-app-mesh-roadmap

AWS App Mesh is a service mesh that you can use with your microservices to manage service to service communication
Apache License 2.0
347 stars 25 forks source link

External requests, containing client certificate, get routed through virtual gateway and fail to reach backend service #457

Open Fmowers opened 1 year ago

Fmowers commented 1 year ago

I require the ability to perform mtls between my backend service (ECS + AppMesh) and an external client outside my network. I have configured a virtual node with a PCA cert to validate client certs with and can confirm it is working with the generated cert + private key. I confirmed this by successfully getting it to work using the virtual gateway as the (cert + private key) tls source. The issue is that when I use the same cert and key that worked between the gateway and node in a curl command I get the following response

upstream connect error or disconnect/reset before headers. 
retried and the latest reset reason: connection failure, 
transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE

When I check the envoy logs I get the following error

[2023-02-22 20:27:14.757][37][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:233] 
[C1406] remote address:10.0.11.203:36680,
TLS error: 268435648:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE

It appears to me that the gateway is not including the certs when it forwards the request to the backend. I also do not see a way to make the gateway listener use tcp instead of http.

Not being able to find a way to allow the certs to survive the gateway I looked to see if I could at least set up mtls between the gateway and the client. Which appeared to be working until I realized I could not make it enforce mtls the same as I could on the node. In other words it didnt matter if I sent a request with or without the client certificates. The gateway just welcomed all through. Unfortunately this defeats the entire reason I need the mtls here in the first place. Due to this being a regulatory restriction. I am not able to use a different solution. It has to be mtls being sent from an external source. Is there anything that I might have missed that could make this work or is it definitely not going to work using the current version? If it cannot currently be done then can you add functionality that will allow this to be done?

srijitm commented 1 year ago

@Fmowers When you setup mTLS between the gateway and the client did you set explicit client name validation in the gateway config?

See: https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-mutual-tls-file-provided#step-2-set-explicit-client-validation-with-an-incorrect-subject-alternative-name