Closed Bfarkiani closed 1 month ago
cc @DavidSchinazi cc @RyanTheOptimist
@jeongseokson could you take a look at this one please?
Hi @Bfarkiani, thank you for reporting the issue. Does sending a CONNECT-UDP request directly to the server-proxy work?
I will try to test the forwarding and terminating proxy two-hop setup myself and see if there's any issue in the current example configs.
Hi @jeongseokson Thank you for your attention. I sent the request to the server. The server uses the same configuration as above (the same as the envoy docs). Envoy server prints the following message
[2024-04-29T20:15:37.750Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 200 - 3875 68976 383 - "-" "-" "c70459b6-454d-45e1-9629-55f97aae6c7f" "www.google.com:443" "142.250.190.36:443"
So, it seems that sending directly to the server works fine.
Thanks for the update. I will test the two proxy setup myself when time permits to see if I can reproduce the issue.
Dear @jeongseokson Have you had any chance to review this matter?
Hi @Bfarkiani, I started looking into this issue last week. I will follow up soon after gathering more information reproducing the issue myself.
I reproduced the setup and tried the same. The issue is due to certificate verification on the forwarding proxy side when establishing a QUIC connection with the upstream terminating proxy. This happens regardless of CONNECT-UDP when a self-signed certificate is used in the terminating proxy.
There is no way to bypass Leaf certificate doesn't match hostname: www.google.com
. You can use a self-signed cert on the proxy server to both downstream and upstream. But your masque client should be configured to expect your proxy server's own cert instead of a google.com cert. auto_sni
will pick the host name from your request url, so it will be google.com. If you explicit config upstream_tls_context.sni
in your masque client config to the SAN in your own cert, you should be able to bypass this error.
Thanks for the reply Dan. But the masque_client has --disable_certificate_verification=true
, and the Leaf certificate doesn't match hostname:
error is from the Envoy (forwading proxy), not from the masque_client. Is there a way to bypass it from the Envoy when it works as a QUIC client (as a forwarding proxy)?
Thanks for the reply Dan. But the masque_client has
--disable_certificate_verification=true
, and theLeaf certificate doesn't match hostname:
error is from the Envoy (forwading proxy), not from the masque_client.
Sorry I'm confused about this comment .
Is below log observed on the client envoy? If so it means it failed handshake with masque client as a server (given tls_server_handshaker
).
[2024-04-25 03:56:25.547][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:974] No hostname indicated in SNI
[2024-04-25 03:56:25.563][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: www.google.com
22063240393008:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:
[2024-04-25T03:56:25.554Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 503 UF 1311 236 9 - "-" "-" "3e1dbc32-dba5-48fa-8069-2065bfecefec" "www.google.com:443" "192.168.170.128:10000"
And this is downstream config. What confuses me is that you said auto_sni
which is in upstream config changes the logging. From [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: 58755145347776:error:1000007d:SSL
to [2024-04-25 03:56:25.563][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: www.google.com 22063240393008:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:
Is there a way to bypass it from the Envoy when it works as a QUIC client (as a forwarding proxy)?
No other than via the sni
config knob.
@Bfarkiani Do you have anything to add regarding the auto_sni
? I think auto_sni
doesn't make sense in this setup because the target hostname (google.com
) is not the hostname of the terminating proxy.
@danzh2010 My understanding is that it's the TLS handshake failure between the forwarding proxy (client Envoy) and the terminating proxy (server Envoy), not between the masque client and the forwarding proxy. Is there a reason you believe it's still a handshake failure between the masque_client and the forwarding envoy?
To try to solve the issue, I set the sni
value in UpstreamTlsContext config of the forwarding proxy to match the self signed certificate's CN of the terminating proxy, but I still got the same Leaf certificate doesn't match hostname:
error from the forwarding proxy side.
I figured out that sni
should match SAN, not CA of the certificate to pass the leaf hostname check test of the Envoy. The test certificates provided have lyft.com
as a SAN. To make the TLS handshake between the proxies work, you should set sni: "lyft.com"
under upstream_tls_context:
.
Please try this @Bfarkiani. If you created your own self-signed certificate, you should specify the SAN there to match the one in the config. I checked other http3 example configs of Envoy and they have sni
field set to avoid this issue I guess. I will update the config soon.
Unfortunately, the CONNECT-UDP request still doesn't go through even after the TLS handshake is successfully done. This one looks like an actual issue related to the HTTP Datagram implementation. I will take a look into it as well, but please try to do it yourself just to make sure we encounter the same issue @Bfarkiani.
Dear @jeongseokson Sorry for my late response. I tried the following configurations after applying your change and also creating a self-signed certificate with SAN of example.com:
client:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: UDP
address: 0.0.0.0
port_value: 10000
udp_listener_config:
quic_options: {}
downstream_socket_config:
prefer_gro: true
filter_chains:
- transport_socket:
name: envoy.transport_sockets.quic
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
downstream_tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/envoy/example.com.crt
private_key:
filename: /etc/envoy/example.com.key
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: HTTP3
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
connect_matcher:
{}
route:
cluster: cluster_0
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
http3_protocol_options:
allow_extended_connect: true
upgrade_configs:
- upgrade_type: CONNECT-UDP
clusters:
- name: cluster_0
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
upstream_http_protocol_options:
auto_sni: true ####<---- I added this but no help
explicit_http_config:
http3_protocol_options:
allow_extended_connect: true
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 192.168.170.128
port_value: 10000
transport_socket:
name: envoy.transport_sockets.quic
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
upstream_tls_context:
# SAN of the certs/servercert.pem used for local testing. Update this if using a different certificate.
sni: example.com
Server:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: UDP
address: 0.0.0.0
port_value: 10000
udp_listener_config:
quic_options: {}
downstream_socket_config:
prefer_gro: true
filter_chains:
- transport_socket:
name: envoy.transport_sockets.quic
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
downstream_tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/envoy/example.com.crt
private_key:
filename: /etc/envoy/example.com.key
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: HTTP3
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
connect_matcher:
{}
route:
cluster: service_google
upgrade_configs:
- upgrade_type: CONNECT-UDP
connect_config:
{}
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
http3_protocol_options:
allow_extended_connect: true
upgrade_configs:
- upgrade_type: CONNECT-UDP
clusters:
- name: service_google
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_google
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: google.com
port_value: 443
My certificate details:
The command I am using for masque client is
./masque_client --disable_certificate_verification=true 127.0.0.1:10000 https://www.google.com
What server envoy prints is
And what happens for packets:
Seems that there is a connection. Although masque client shows nothing.
As you see also server envoy prints "No hostname indicated in SNI" even with auto_sni.
@Bfarkiani, I think you shouldn't use auto_sni
in this case. It sets the SNI value same as the host/authority, which isn't correct in this forwarding proxy scenario.
The only change I made was to add sni
in the upstream tls context of the forwading proxy. I added it to match the SAN in the certificate of the terminating proxy, as you can see in the PR: https://github.com/envoyproxy/envoy/pull/34359/files
@jeongseokson I changed my client to chromium masque https://www.chromium.org/quic/playing-with-quic/ and tried the same configuration without auto_sni to connect to a local website. Still Failure with this error at client.
@Bfarkiani I think we've reached the same actual error in the existing forwarding proxy mode I mentioned in a previous comment. Fixing this would probably require some modification to the existing HTTP Datagram handler unfortunately. I will create a separate issue for tracking this as this isn't a configuration issue.
@jeongseokson Thank you.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
@RyanTheOptimist could you reopen this issue and mark no-stale-bot please? I don't think it's fully resolved yet
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
Just want to clarify that the issue raised in this post (as the title says) is about the incorrect example config (No SNI specified), which has been resolved. We uncovered the other issue in the CONNECT-UDP forwarding mode in the process, and I created #34836 for that. I think we can mark this issue resolved and use #34836 to track the issue separately, which I am currently working on.
That makes sense, thanks for the details. OK let's close this issue once https://github.com/envoyproxy/envoy/issues/34836 is reopened
If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via emailing envoy-security@googlegroups.com where the issue will be triaged appropriately.
Title: CONNECT-UDP example configuration error Description: I am trying to test the configuration provided in https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades#connect-udp-support. I am also using quiche masque_client https://github.com/google/quiche/tree/main/quiche/quic/masque. Also I am using envoy:dev latest container. The client side proxy configuration is as below:
Server side proxy (192.168.170.128) that terminates connect-udp and forward request to google is as follows:
And the command for masque client is
./masque_client --disable_certificate_verification=true 127.0.0.1:10000 https://www.google.com
It fails. The output of client envoy is
The server envoy output is
The client and server envoy proxies both use the same self-signed certificate, and the configuration is the same as provided in the website. If I enable auto_sni in client side :
I still get the following error
I also changed server proxy to the follow and nothing changed:
Thank you for checking this.