TCP logging is not there on ambassador

[kannanvr]

Describe the bug When we have configured HTTP/HTTPS, we can see all the logs related to the incoming connection on ambassador pod. But For TCP connection, we could not get the logs at all.

To Reproduce Steps to reproduce the behavior: Folowing is my ambassador deployment container yaml

      - name: ambassador
        image: quay.io/datawire/ambassador:1.3.0
        imagePullPolicy: "IfNotPresent"
        resources:
          limits:
            cpu: 1
            memory: 400Mi
          requests:
            cpu: 200m
            memory: 100Mi
        env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: APP_LOG_LEVEL
          value: trace
        - name: AES_LOG_LEVEL
          value: trace
        - name: SCOUT_DISABLE
          value: "1"

I have enabled AES_LOG_LEVEL and APP_LOG_LEVEL. But when i am adding the TCPMAPPING on a annotations of service , TCP connection is working fine. But We need logs to debug the connection.

apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: TCPMapping
      name: postgres-1
      port: 5633
      service: postges-ingress.postges1:5432

We are expecting the TCP log when we establish the TCP connection through Ambassador

Expected behavior We are expecting the TCP log when we establish the TCP connection through Ambassador

Versions (please complete the following information):

• Ambassador: 1.3.0
  • Kubernetes Version 1.15.3

[cakuros]

So, I tested a bit with a TCPMapping that looks like this:

---
apiVersion: getambassador.io/v2
kind: TCPMapping
metadata:
  name: default-test
spec:
  port: 2222
  service: quote:80

I was finding in my logs (I have set to debug) that there are entries that look like this:

time="2020-09-16 16:11:58" level=debug msg="[gRPC] HTTP/1.1 GET admin.REDACTED.net /backend/" func="github.com/datawire/apro/cmd/amb-sidecar/filters/handler.(*FilterMux).Filter" file="github.com/datawire/apro/cmd/amb-sidecar/filters/handler/handler.go:108" MAIN=http REQUEST_ID=15892106509584408014 SUB=http-handler
time="2020-09-16 16:11:58" level=debug msg="[gRPC] *filterapi.HTTPRequestModification : 0 headers (124.374µs)" func=github.com/datawire/apro/cmd/amb-sidecar/filters/handler.logResponse file="github.com/datawire/apro/cmd/amb-sidecar/filters/handler/handler.go:68" MAIN=http REQUEST_ID=15892106509584408014 SUB=http-handler

that were corresponding to the requests I was sending via curl https://admin.REDACTED.net:2222

[kannanvr]

@cakuros , Thanks for your reply. Can you please share your ambassador deployment yaml file. I think my deployment might be wrong. I will also check it.

Thanks,

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bb3333bb]

upgrade to v1.9.0 using manifest from https://www.getambassador.io/docs/latest/topics/install/install-ambassador-oss/ and still see no log from tcpmapping When i cat envoy/envoy.json, just see for tcp

 {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 4444,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "name": "envoy.filters.network.tcp_proxy",
                                "typed_config": {
                                    "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                                    "stat_prefix": "ingress_tcp_4444",
                                    "weighted_clusters": {
                                        "clusters": [
                                            {
                                                "name": xxxxxxxxxxxxxxxxxxxxx_ambassador",
                                                "weight": 100
                                            }
                                        ]
                                    }
                                }
                            }
                        ]
                    }
                ],
                "name": "listener-0.0.0.0-4444"

not see any "access_log" fields as http Ambassador version: 1.9.0 Kubernetes version: 1.19.0

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[woz5999]

i'm also not able to see any tcp logs

[notjames]

I think the ask here should be if we could get the log_format changed a bit to distinguish between HTTP traffic and non-HTTP traffic easier and for TLS termination, something that screams so (like TLS passthrough or something) would be nice.

29697   │             "filter_chain_match": {
29698   │               "server_names": [
29699   │                 "stunnel.<REDACTED>.io"
29700   │               ],
29701   │               "transport_protocol": "tls"
29702   │             },
29703   │             "filters": [
29704   │               {
29705   │                 "name": "envoy.filters.network.http_connection_manager",
29706   │                 "typed_config": {
29707   │                   "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
29708   │                   "access_log": [
29709   │                     {
29710   │                       "name": "envoy.access_loggers.file",
29711   │                       "typed_config": {
29712   │                         "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
29713   │                         "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORW
        │ ARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
29714   │                         "path": "/dev/fd/1"
29715   │                       }
29716   │                     }
29717   │                   ],

It's a little difficult for me to be more specific right now because I'm still trying to get our service to work through ambassador, which is more difficult to troubleshoot because the log_format is seemingly similar or the same as the rest of the HTTP traffic.

[juanjoku]

I am also interested in obtaining logs related to Envoy tcp listeners (tcpmappings).

[juanjoku]

Hello,

I'm testing the new Ambassador v3.5, and access-logs are not available for TCP requests (TCPMappings).

Are you planning to include it in future versions?

Thanks!!

[dgaffuri]

Any news on this topic? Still not present in 3.9.1

Thanks