search

istio / istio

Connect, secure, control, and observe services.
https://istio.io
Apache License 2.0
35.89k stars 7.74k forks source link

When kubernetes cluster deploy statefulset redis cluster, the application cannot connect to external redis service #24841

Closed sundongmin closed 4 years ago

sundongmin commented 4 years ago

Bug description In kubernetes, I hava two nameapace, a called dev, another called citest. I hava a java application run in dev, and need to connect to external redis cluster(use Amazon ElastiCache), when citest has a statefulset redis cluster(no sidecar), then java applicatioin cannot connect to external redis cluster

Expected behavior

  1. Successful connect to external redis cluster
  2. From the below proxy config, I see the Istio create a listener on 0.0.0.0_6379, but I don't understand why all the tcp traffic route to the redis-cluster.citest.svc.cluster.local, Can anyone tell me why?

Steps to reproduce the bug

  1. deploy a java application, and use external redis services
  2. in another namespace(in my case, is citest), deploy a statefulset redis cluster 
    apiVersion: v1
kind: Service
metadata:
name: redis-cluster
labels:
app: redis-cluster
spec:
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
clusterIP: None
selector:
app: redis-cluster

    Here's log from that time

    • the application log(only paste one line, other same) 
      2020-06-20 22:42:08.208  INFO 6 --- [llEventLoop-4-2] i.l.core.protocol.ReconnectionHandler    : Reconnected to xxx.xxx.xxx.xxx.cache.amazonaws.com.cn:6379
    • the istio-proxy log(only paste one line, other same) 
      {
"authority": "-",
"bytes_received": "0",
"bytes_sent": "0",
"downstream_local_address": "10.3.24.247:6379",
"downstream_remote_address": "10.244.1.129:50394",
"duration": "9999",
"istio_policy_status": "-",
"method": "-",
"path": "-",
"protocol": "-",
"request_id": "-",
"requested_server_name": "-",
"response_code": "0",
"response_flags": "UF,URX",
"route_name": "-",
"start_time": "2020-06-20T14:46:33.512Z",
"upstream_cluster": "outbound|6379||redis-cluster.citest.svc.cluster.local", 
"upstream_host": "10.3.24.247:6379",
"upstream_local_address": "-",
"upstream_service_time": "-",
"upstream_transport_failure_reason": "-",
"user_agent": "-",
"x_forwarded_for": "-"
}
    • the envoy listener config 
      istioctl pc listener org-api-599588fb86-l5fzn -n dev --port 6379 --address 0.0.0.0 -o json
[
{
"name": "0.0.0.0_6379",
"address": {
    "socketAddress": {
        "address": "0.0.0.0",
        "portValue": 6379
    }
},
"filterChains": [
    {
        "filters": [
            {
                "name": "istio.stats",
                "typedConfig": {
                    "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                    "typeUrl": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                    "value": {
                        "config": {
                            "configuration": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\"\n}\n",
                            "root_id": "stats_outbound",
                            "vm_config": {
                                "code": {
                                    "local": {
                                        "inline_string": "envoy.wasm.stats"
                                    }
                                },
                                "runtime": "envoy.wasm.runtime.null",
                                "vm_id": "tcp_stats_outbound"
                            }
                        }
                    }
                }
            },
            {
                "name": "envoy.tcp_proxy",
                "typedConfig": {
                    "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                    "statPrefix": "outbound|6379||redis-cluster.citest.svc.cluster.local",
                    "cluster": "outbound|6379||redis-cluster.citest.svc.cluster.local",
                    "accessLog": [
                        {
                            "name": "envoy.file_access_log",
                            "typedConfig": {
                                "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                "path": "/dev/stdout",
                                "jsonFormat": {
                                    "authority": "%REQ(:AUTHORITY)%",
                                    "bytes_received": "%BYTES_RECEIVED%",
                                    "bytes_sent": "%BYTES_SENT%",
                                    "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
                                    "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
                                    "duration": "%DURATION%",
                                    "istio_policy_status": "%DYNAMIC_METADATA(istio.mixer:status)%",
                                    "method": "%REQ(:METHOD)%",
                                    "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
                                    "protocol": "%PROTOCOL%",
                                    "request_id": "%REQ(X-REQUEST-ID)%",
                                    "requested_server_name": "%REQUESTED_SERVER_NAME%",
                                    "response_code": "%RESPONSE_CODE%",
                                    "response_flags": "%RESPONSE_FLAGS%",
                                    "route_name": "%ROUTE_NAME%",
                                    "start_time": "%START_TIME%",
                                    "upstream_cluster": "%UPSTREAM_CLUSTER%",
                                    "upstream_host": "%UPSTREAM_HOST%",
                                    "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
                                    "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
                                    "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
                                    "user_agent": "%REQ(USER-AGENT)%",
                                    "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
                                }
                            }
                        }
                    ]
                }
            }
        ]
    },
    {
        "filterChainMatch": {
            "applicationProtocols": [
                "http/1.0",
                "http/1.1",
                "h2c"
            ]
        },
        "filters": [
            {
                "name": "envoy.http_connection_manager",
                "typedConfig": {
                    "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                    "statPrefix": "outbound_0.0.0.0_6379",
                    "rds": {
                        "configSource": {
                            "ads": {}
                        },
                        "routeConfigName": "6379"
                    },
                    "httpFilters": [
                        {
                            "name": "istio.metadata_exchange",
                            "typedConfig": {
                                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                "typeUrl": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                                "value": {
                                    "config": {
                                        "configuration": "{}\n",
                                        "vm_config": {
                                            "code": {
                                                "local": {
                                                    "inline_string": "envoy.wasm.metadata_exchange"
                                                }
                                            },
                                            "runtime": "envoy.wasm.runtime.null"
                                        }
                                    }
                                }
                            }
                        },
                        {
                            "name": "istio.alpn",
                            "typedConfig": {
                                "@type": "type.googleapis.com/istio.envoy.config.filter.http.alpn.v2alpha1.FilterConfig",
                                "alpnOverride": [
                                    {
                                        "alpnOverride": [
                                            "istio-http/1.0",
                                            "istio"
                                        ]
                                    },
                                    {
                                        "upstreamProtocol": "HTTP11",
                                        "alpnOverride": [
                                            "istio-http/1.1",
                                            "istio"
                                        ]
                                    },
                                    {
                                        "upstreamProtocol": "HTTP2",
                                        "alpnOverride": [
                                            "istio-h2",
                                            "istio"
                                        ]
                                    }
                                ]
                            }
                        },
                        {
                            "name": "envoy.cors",
                            "typedConfig": {
                                "@type": "type.googleapis.com/envoy.config.filter.http.cors.v2.Cors"
                            }
                        },
                        {
                            "name": "envoy.fault",
                            "typedConfig": {
                                "@type": "type.googleapis.com/envoy.config.filter.http.fault.v2.HTTPFault"
                            }
                        },
                        {
                            "name": "istio.stats",
                            "typedConfig": {
                                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                "typeUrl": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                                "value": {
                                    "config": {
                                        "configuration": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\"\n}\n",
                                        "root_id": "stats_outbound",
                                        "vm_config": {
                                            "code": {
                                                "local": {
                                                    "inline_string": "envoy.wasm.stats"
                                                }
                                            },
                                            "runtime": "envoy.wasm.runtime.null",
                                            "vm_id": "stats_outbound"
                                        }
                                    }
                                }
                            }
                        },
                        {
                            "name": "envoy.router",
                            "typedConfig": {
                                "@type": "type.googleapis.com/envoy.config.filter.http.router.v2.Router"
                            }
                        }
                    ],
                    "tracing": {
                        "clientSampling": {
                            "value": 100
                        },
                        "randomSampling": {
                            "value": 100
                        },
                        "overallSampling": {
                            "value": 100
                        }
                    },
                    "streamIdleTimeout": "0s",
                    "accessLog": [
                        {
                            "name": "envoy.file_access_log",
                            "typedConfig": {
                                "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                "path": "/dev/stdout",
                                "jsonFormat": {
                                    "authority": "%REQ(:AUTHORITY)%",
                                    "bytes_received": "%BYTES_RECEIVED%",
                                    "bytes_sent": "%BYTES_SENT%",
                                    "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
                                    "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
                                    "duration": "%DURATION%",
                                    "istio_policy_status": "%DYNAMIC_METADATA(istio.mixer:status)%",
                                    "method": "%REQ(:METHOD)%",
                                    "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
                                    "protocol": "%PROTOCOL%",
                                    "request_id": "%REQ(X-REQUEST-ID)%",
                                    "requested_server_name": "%REQUESTED_SERVER_NAME%",
                                    "response_code": "%RESPONSE_CODE%",
                                    "response_flags": "%RESPONSE_FLAGS%",
                                    "route_name": "%ROUTE_NAME%",
                                    "start_time": "%START_TIME%",
                                    "upstream_cluster": "%UPSTREAM_CLUSTER%",
                                    "upstream_host": "%UPSTREAM_HOST%",
                                    "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
                                    "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
                                    "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
                                    "user_agent": "%REQ(USER-AGENT)%",
                                    "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
                                }
                            }
                        }
                    ],
                    "useRemoteAddress": false,
                    "generateRequestId": true,
                    "upgradeConfigs": [
                        {
                            "upgradeType": "websocket"
                        }
                    ],
                    "normalizePath": true
                }
            }
        ]
    }
],
"deprecatedV1": {
    "bindToPort": false
},
"listenerFilters": [
    {
        "name": "envoy.listener.tls_inspector",
        "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.filter.listener.tls_inspector.v2.TlsInspector"
        }
    },
    {
        "name": "envoy.listener.http_inspector",
        "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.filter.listener.http_inspector.v2.HttpInspector"
        }
    }
],
"listenerFiltersTimeout": "0.100s",
"continueOnListenerFiltersTimeout": true,
"trafficDirection": "OUTBOUND"
}
]

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

# istio version
client version: 1.6.2
control plane version: 1.6.2
data plane version: 1.6.2

# kubernetes version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"1bea6c00a7055edef03f1d4bb58b773fa8917f11", GitTreeState:"clean", BuildDate:"2020-02-11T20:13:57Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"1bea6c00a7055edef03f1d4bb58b773fa8917f11", GitTreeState:"clean", BuildDate:"2020-02-11T20:05:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? istioctl

Environment where bug was observed (cloud vendor, OS, etc) kubernetes clusters with kubeadm on aws ec2

hzxuzhonghu commented 4 years ago

If you run istioctl pc listener org-api-599588fb86-l5fzn -n dev --port 6379, you can see istio generates one listeners per redis instance.

It seems you met the known issue with automtls. For more context #21964

The workaround is disable mtls explicitly with destinationrule.

DestinationRule.trafficPolicy.Tls == DISABLE https://github.com/istio/istio/issues/21964#issuecomment-596382298

howardjohn commented 4 years ago

Closing as a duplicate of https://github.com/istio/istio/issues/21964 as this seems the same issue and has not had any updates for quite some time