tls issue on GKE cluster

[aviramha]

Bug Description

mirrord fails after first TLS message

Steps to Reproduce

n/a

Backtrace

No response

Relevant Logs

2023-08-24T20:06:21.488189Z DEBUG HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: kube_client::client::builder: requesting
2023-08-24T20:06:21.488205Z TRACE HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: hyper::client::pool: checkout waiting for idle connection: ("https", redacted)
2023-08-24T20:06:21.488227Z TRACE HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("redacted"), port=None
2023-08-24T20:06:21.488235Z DEBUG HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: hyper::client::connect::http: connecting to redacted:443
2023-08-24T20:06:21.488944Z DEBUG HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: hyper::client::connect::http: connected to redacted:443
2023-08-24T20:06:21.489009Z DEBUG HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: rustls::client::hs: No cached session for IpAddress(redacted)    
2023-08-24T20:06:21.489054Z DEBUG HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: rustls::client::hs: Not resuming any session    
2023-08-24T20:06:21.489099Z TRACE HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: rustls::client::hs: Sending ClientHello Message {
    version: TLSv1_0,
    payload: Handshake {
        parsed: HandshakeMessagePayload {
            typ: ClientHello,
            payload: ClientHello(
                ClientHelloPayload {
                    client_version: TLSv1_2,
                    random: 0bfce26ebbfec33805d25c063d8269aa908a16e648b27c352779d6917079511c,
                    session_id: 6d25edbbebc5a84e864611461ab0ad559a9236ee30dd05f68eb1c26088d6d57f,
                    cipher_suites: [
                        TLS13_AES_256_GCM_SHA384,
                        TLS13_AES_128_GCM_SHA256,
                        TLS13_CHACHA20_POLY1305_SHA256,
                        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                        TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
                        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
                        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
                        TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
                        TLS_EMPTY_RENEGOTIATION_INFO_SCSV,
                    ],
                    compression_methods: [
                        Null,
                    ],
                    extensions: [
                        SupportedVersions(
                            [
                                TLSv1_3,
                                TLSv1_2,
                            ],
                        ),
                        ECPointFormats(
                            [
                                Uncompressed,
                            ],
                        ),
                        NamedGroups(
                            [
                                X25519,
                                secp256r1,
                                secp384r1,
                            ],
                        ),
                        SignatureAlgorithms(
                            [
                                ECDSA_NISTP384_SHA384,
                                ECDSA_NISTP256_SHA256,
                                ED25519,
                                RSA_PSS_SHA512,
                                RSA_PSS_SHA384,
                                RSA_PSS_SHA256,
                                RSA_PKCS1_SHA512,
                                RSA_PKCS1_SHA384,
                                RSA_PKCS1_SHA256,
                            ],
                        ),
                        ExtendedMasterSecretRequest,
                        CertificateStatusRequest(
                            OCSP(
                                OCSPCertificateStatusRequest {
                                    responder_ids: [],
                                    extensions: ,
                                },
                            ),
                        ),
                        SignedCertificateTimestampRequest,
                        KeyShare(
                            [
                                KeyShareEntry {
                                    group: X25519,
                                    payload: redacted,
                                },
                            ],
                        ),
                        PresharedKeyModes(
                            [
                                PSK_DHE_KE,
                            ],
                        ),
                        SessionTicket(
                            Request,
                        ),
                    ],
                },
            ),
        },
        encoded: redacted,
    },
}    
2023-08-24T20:06:21.538077Z TRACE HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client"}: hyper::client::pool: checkout dropped for ("https", redacted)
2023-08-24T20:06:21.538143Z ERROR HTTP{http.method=GET http.url=https://redacted/api/v1/namespaces/test5-us/pods/services-5b95bb899-b4767 otel.name="get" otel.kind="client" otel.status_code="ERROR"}: kube_client::client::builder: failed with error error trying to connect: tls handshake eof
2023-08-24T20:06:21.538226Z TRACE tower::buffer::worker: worker polling for next message
2023-08-24T20:06:21.538249Z TRACE tower::buffer::worker: buffer already closed
Error:   × Create agent failed. KubeError(
  │     HyperError(
  │         hyper::Error(
  │             Connect,
  │             Custom {
  │                 kind: Other,
  │                 error: Custom {
  │                     kind: UnexpectedEof,
  │                     error: "tls handshake eof",
  │                 },
  │             },
  │         ),
  │     ),
  │ )

### Your operating system and version

n/a

### Local process

n/a

### Local process version

_No response_

### Additional Info

_No response_

[aviramha]

Opened issue upstream https://github.com/rustls/rustls/issues/1417

[aviramha]

Doesn't seem to be upstream. Seems most likely user has a local firewall denying access for some reason. Probably application level because kubectl get pods works for them

[aviramha]

From user on Discord:

I'll run a tcpdump and get that out to you. Re: your troubleshooting questions: I'm running the latest version of mirrord.Don't have a local firewall, but do have netskope running on my laptop which could be doing some weird things with SSL. I'm going to try and compare by testing on my own personal computer in case it is that. I don't think we have anything on the cluster side, given that it's GKE and the SSL cert/IP endpoint is owned by them.

[aviramha]

Netskope didn't trust SSL without hostname (IP) so kubectl also was excluded.