Connection closed when connecting to TCP services

[senthilrameshjv]

What steps did you take and what happened: [A clear and concise description of what the bug is.] I am working on implementing HTTPProxy for TCP connections in an on-prem solution with no load balancer. Your examples have helped me to get started, however, I find the instructions are not clear when it comes to TCP in bare metal situations with no load balancer. I was not able to understand how to apply the host network instructions in testing HTTP proxy

I applied kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

and deployed kuard application as well.

my DAG application shows listeners:443 -> service:service-port

I can connect to service:service-port and it reaches my application, but when I connect to fqdn:443, I get connection closed. I defined fqdn hostname to be 127.0.0.1 or also the 192.168.. address of my VM where I am trying to apply these. Both didn't work.

I run a single node kubernetes cluster and my services/pods are deployed directly in master node

What did you expect to happen:

I expected the connection to be forwarded to service and then forwarded to my pod.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

A sample yaml I use.

apiVersion: v1
kind: Service
metadata:
  name: example-service
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
    - name: svc-jdbc-non-rmi
      port: 8996
      targetPort: jdbc-non-rmi
      protocol: TCP
    - name: svc-webservices
      port: 8090
      targetPort: 9090
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: webcontainer-proxy
spec:
  virtualhost:
    fqdn: example-webcontainer     
  routes:
    - conditions: 
      - prefix: / 
      services:
        - name: example-service
          port: 8090
---              
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: jdbc-non-rmi-proxy
spec:
  virtualhost:
    fqdn: example-jdbc-non-rmi     
    tls:
      #secretName: default/default-ssl-certificate
      #secretName: example-ssl-certificate
      passthrough: true         
  tcpproxy:
    services:
    - name: example-service
      port: 8996
---

Contour-DAG:

Environment:

• Contour version: latest
• Kubernetes version: (use kubectl version): Client Version: v1.18.3 Server Version: v1.18.3
• Kubernetes installer & version:
• Cloud provider or hardware configuration: On Prem
• OS (e.g. from /etc/os-release): Ubuntu 20.04 LTS running in Virtual box with Windows 10 host

[youngnick]

Thanks for the issue @senthilrameshjv.

Firtly, could you reformat your YAML using the ``` pre-formatted markers? It's tricky to check the YAML for errors if it's not formatted correctly.

That said, assuming the spacing is correct for the YAML, since the DG is generating correctly, it looks like the TCP Proxy should be working.

A couple of questions to check basic things:

• Can you reach the example-webcontainer host successfully?
• If you use openssl s_client -connect <ip>:443 to connect to the TCP forwarded service, do you see any certificates at all? If not, you may not be connecting to what you think; if Envoy has the config, and you connect to port 443 on the Envoy, everything should work.

[senthilrameshjv]

Thank you @youngnick. I edited previous yaml.

Yes I can reach to example-webcontainer. That makes me believe that port 80 is listening and 443 is not. As I am using tls pass through and my client and the application has the certificates required, should there be any certificates in the ingress layer?

Attaching the output of OpenSSL command.

openssl s_client -connect example-jdbc-non-rmi:443

CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

My updated DAG. I also included tls pass through now for example-webcontainer

[youngnick]

Hi @senthilrameshjv, can you check the status on the HTTPProxy once it's configured? In addition, can you please check the Envoy access logs and confirm put some relevant lines in here? We should be able to see some log lines telling us that the TCP proxy has been hit, I think.

You are correct, you shouldn't need certificates in the TCP Proxy config, since you are doing passthrough.

The openssl output does indicate that you're not getting a TLS connection negotiation though, so we need to find out what's happening to the request.

[senthilrameshjv]

I also read the readme.md and modified the 03-envoy and 02-service-envoy yaml files to include dns policy, host network and load balancing changes. Still the same connection issues.

Here are my httpproxies

NAMESPACE   NAME                 FQDN                   TLS SECRET   STATUS    STATUS DESCRIPTION
default     factory-proxy        example-factory                     valid     valid HTTPProxy
default     jdbc-non-rmi-proxy   example-jdbc-non-rmi                valid     valid HTTPProxy
default     jdbc-proxy           example-jdbc                        valid     valid HTTPProxy
default     kuard                kuard.local                         valid     valid HTTPProxy
default     tls-passthrough      kahttp.localdomain                  invalid   tcpproxy: service default/kahttp/443: not found
default     webcontainer-proxy   example-webcontainer                valid     valid HTTPProxy

Ignore the invalid one as that is the one i used following this

Any reference on how to check the logs? I saw the documentation on forwarding to 9001. Is that something i should check?

[youngnick]

The Envoy logs are written to stdout in the container, you can use kubectl logs <podname> to get them.

Great that everything is valid, I was pretty sure that would be the case from the DAG diagram, but good to check.

[senthilrameshjv]

Thank you, but I dont see much info with logs

kubectl logs envoy-5rcrb -n projectcontour -c envoy-initconfig
kubectl logs envoy-5rcrb -n projectcontour -c shutdown-manager
time="2020-06-19T03:03:08Z" level=info msg="started envoy shutdown manager" context=shutdown-manager

Can you help me about the hostnetwork configuration when there is no load balancer available. It is possible some configuration is missing?

[youngnick]

With respect to the logs, sorry, you need to check the envoy container's logs. I should have mentioned that.

With the hostnetwork config, yes, it's definitely possible. I asked about connecting the non-passthrough service because I assumed if you can connect to that, then you can connect through Envoy, so the hostnetwork config is probably working.

To be clear, you should be connecting to the example-webcontainer service using http://example-webcontainer/, as in:

curl http://example-webcontainer/

And you should be connecting to the passthrough service with:

curl https://example-jdbc-non-rmi/

Both of those names should resolve to the IP address that is the 'outside' of Envoy, the address that has ports 80 and 443 forwarded to Envoy.

In addition, the certificate that is presented by example-jdbc-non-rmi should have that as a Subject Alternative Name in the certificate.

[senthilrameshjv]

Let me check the container logs. Meanwhile, here is the response for https curl

curl --cacert cacert.pem https://example-jdbc-non-rmi/
curl: (52) Empty reply from server

I had to use additional parameters, as its self signed certificate.

Also, the tcp connections i make are from a Java client to a Java application. I was trying to connect using //example-jdbc-non-rmi:443 address, where FQDN is pointed to 127.0.0.1 (my cluster is a single node master-only setup with pods deployed in master)

Let me know your thoughts.

[youngnick]

Is it expected that making a request to / would produce an empty reply? Should the backend server have responded with something? The thing we need to determine here is whether Contour is misconfiguring Envoy so that the connection is not passing through somehow (in which case Envoy would terminate it), or if the connection is arriving at the backend service and not behaving as expected somehow.

Does the Java application's transport use HTTPS, or some other TLS-secured protocol?

[senthilrameshjv]

Sorry for the delay.

I am able to reach out directly inside the cluster to the respective service IP addresses. I think there is something broken when forwarding from ingress to the service.

[senthilrameshjv]

Chiming in with an update. I was able to do a TCP forwarding with Nginx-ingress controller with its TCP Config map and adding ports to its services following a guide like this

I am not able to understand what I will have to include in the envoy/contour yamls. I added the hostnetwork and dns policy following 'Running without Kubernetes load balancer' in Deploy Options and I see that the contour file already has the envoy ports defined.

Alternatively, to check if any TCP connections are working, if you can help me to deploy a sample TCP application like this, it will be great. I was able to deploy this in nginx-ingress controller but I wonder if I can do using ProjectContour as Contour provides me more flexibility using virtual FQDNs.

[stevesloka]

Hey @senthilrameshjv does your app have a TLS certificate? So if you port-forward to that app you can access it over TLS?

I ask because in your example above, you have Spec.VirtualHost.Tls.Passthrough: true, meaning the upstream service is intended to terminate TLS at the backend pod.

If that's not the case, then you'd need to have TLS secrets on the Spec.VirtualHost.Tls section to terminate TLS at Envoy, then proxy to the upstream properly.

Today, Contour only proxies TCP over SNI, meaning, there must be TLS either terminating at Envoy or the backend pod.

[senthilrameshjv]

@stevesloka Yes mine has TLS enabled. I used passthrough: true And both my client and the backend server has the required certificates. I remember sometime back, I used a port forward and was able to access it using loopback address, but not from external machines using the machine's public IP address.

[moderation]

Wondering if this TCP bug fixed in Envoy 1.15.0 is a contributing factor to this issue? https://github.com/envoyproxy/envoy/issues/12197#issuecomment-661649308

[youngnick]

@moderation I don't think it's possible to tell at this point.

Rereading this ticket, it seems that @senthilrameshjv is using the TCPProxy forwarding for a TLS-secured, but non-HTTP protocol. Please correct me if I'm wrong, but I don't think JDBC uses HTTP as a transport.

We've built the TCP Proxy support into HTTPProxy with the main intent of having people use it for passing through HTTPS connections to a backend service that wants to terminate its own SSL.

For doing TCPProxy for non-HTTP protocols, we haven't tested this at all. It's technically possible, I guess, but I have no idea about edge cases. The best I can do is say the things that I do know Contour and Envoy will require in order to route the requests properly.

Firstly, Contour will need the Service that you're terminating on to have an annotation of projectcontour.io/upstream-protocol: tls. This allows Contour to configure Envoy with the correct forwarding settings for the TLS connection. (Other options are "h2", or "h2c", meaning HTTP/2 and HTTP/2 in the clear respectively).

Secondly, you will need to enable passthrough: true in the tls block inside the virtualhost block in the relevant HTTPProxy.

Thirdly, the backend service will need to supply valid serving certs for the domain mentioned in the virtualhost.fqdn field.

Fourth, the client that connects to the TLS passthrough service must send an SNI for the domain mentioned in the virtualhost.fqdn field. The fallback certificate support that Contour allows will only work for HTTP services or services that terminate TLS at Envoy, as it requires inspection of the Host header. For passthrough services or services that do not use HTTP, it will not work.

I haven't had a chance to set up something to test all of this, as it's quite far outside our usual rubric - being a HTTP ingress controller.

[youngnick]

@senthilrameshjv, any updates on this issue?

[alparslanavci]

Guys, I just would like to verify that TCPProxy for non-HTTP protocols works for both TLS session passthrough and proxying. I managed to set up two Hazelcast clusters to replicate data (which is performed by TCP/IP using TLS) via Contour ingress.

@youngnick, your previous comment helped a lot, thank you.

[youngnick]

Thanks for that @alparslanavci!

[senthilrameshjv]

@youngnick No, we had to use different frameworks instead of projectcontour.

[youngnick]

Looking back at this issue, I think the thing we need to do here is clarify the requirements around using TCPProxy, as per my earlier comment. I'll add the doc-impact and help wanted labels to indicate that we need a docs update here.

I don't think it's a good place for doc writers to get started though, so no good first issue.

[youngnick]

I'm sorry @senthilrameshjv, I forgot to thank you for all the great reporting you did on this issue. I'm also sorry that Contour didn't turn out to be the right fit for you, hopefully some documentation updates should help others in the future.

[plaisted]

I'm having some trouble with this as well and am beginning to think contour doesn't really fit for non-http traffic.

Snippets setup below from discussion above:

# snippet
apiVersion: projectcontour.io/v1
kind: HTTPProxy
spec:
  virtualhost:
    fqdn: url.example.com
    tls:
      passthrough: true
  tcpproxy:
    services:
      - name: my-service
        port: 4321

# on service my-service
  annotations:
    projectcontour.io/upstream-protocol: tls

Using a simple kubectl port-forward I am able to connect to the pods using TLS over TCP without issue (using cert valid for both localhost and the fqdn). Trying to connect on 443 or 80 of the fqdn does not work through contour (get 'server did not properly respond after a period of time'). A second contour proxy on the same service (with different port) works fine using http. I don't see anything in logs from contour or envoy to give me any hint.

edit: My issues had to do with NATS which sends some data over normal TCP before upgrading to TLS. This doesn't work with contour as it needs SNI for routing.

[sachinshakya507]

TCP and UDP services can be accessed in nginx ingress-controller like this Is it still not possible for contour ?

[dimashenkov]

@plaisted Hi, did you found solution , Im fighting with NATS exposure with contour too?