Istio Egress gateway for TCP traffic port 22

bhat39a commented 3 months ago

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

Hi Team,

We are getting the below error in the istio-proxy sidecar container, we have egress for wildcard and its working fine but traffic for port 22 is not being routed to egressgateway, below is our configuration file. I am not sure what are we missing in here.

Error From istio-proxy sidecar :

[2024-06-18T13:17:50.593Z] "- - -" 0 - - - "-" 2313 7962 130 - "-" "-" "-" "-" “xx.xx.xx.xx:8443" outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local xx.xx.xx.xx:46114 xx.xx.xx.xx:443 xx.xx.xx.xx:47368 ec2.eu-west-1.amazonaws.com - [2024-06-18T13:17:50.729Z] "- - -" 0 - - - "-" 2306 7960 89 - "-" "-" "-" "-" “xx.xx.xx.xx:8443" outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local xx.xx.xx.xx:46120 xx.xx.xx.xx:443 xx.xx.xx.xx:50828 ec2.eu-west-1.amazonaws.com - [2024-06-18T13:17:50.821Z] "- - -" 0 NC - - "-" 0 0 0 - "-" "-" "-" "-" "-" - - xx.xx.xx.xx:22 xx.xx.xx.xx:33264 - - [2024-06-18T13:17:50.821Z] "- - -" 0 NC - - "-" 0 0 0 - "-" "-" "-" "-" "-" - - xx.xx.xx.xx:22 xx.xx.xx.xx:33266 - - [2024-06-18T13:17:17.231Z] "- - -" 0 - - - "-" 105136 93604 89986 - "-" "-" "-" "-" “xx.xx.xx.xx:8443" outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local xx.xx.xx.xx:47366 xx.xx.xx.xx:443 xx.xx.xx.xx:47536 xx.xx.xx.com -

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: se-port-22
  namespace: istio-system
spec:
  hosts:
  - "example.com"
  addresses:
  - 10.0.0.0/8
  ports:
  - number: 22
    name: tcp
    protocol: TCP
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 22
      name: tcp
      protocol: TCP
    hosts:
    - "example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-tcp
  namespace: istio-system
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tcp-through-egress-gateway
  namespace: istio-system
spec:
  hosts:
  - "example.com"
  tcp:
  - match:
    - port: 22
    route:
    - destination:
        host: "istio-egressgateway.istio-system.svc.cluster.local"
        port:
          number: 22

Version

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-eks-036c24b

Additional Information

No response

howardjohn commented 3 months ago

outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local implies its going to subset=wildcard. That is not defined anywhere in your config in the issue. I am fairly confident you have some config that is not in the issue causing this.,

bhat39a commented 3 months ago

outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local implies its going to subset=wildcard. That is not defined anywhere in your config in the issue. I am fairly confident you have some config that is not in the issue causing this.,

yes we have a separate config for wildcard hosts so that entry is working fine. Traffic to *.com is getting redirected to a different egress gateway but traffic routing for TCP port 22 is not working.

howardjohn commented 3 months ago

Ah sorry, got it. Do you have port 22 defined on the gateway Service?

bhat39a commented 3 months ago

Ah sorry, got it. Do you have port 22 defined on the gateway Service?

do you mean this. Posted the entire config file above.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 22
      name: tcp
      protocol: TCP
    hosts:
    - "example.com"

howardjohn commented 3 months ago

That just configures the GW, it still needs to be in the Service object as well

bhat39a commented 3 months ago

That just configures the GW, it still needs to be in the Service object as well

Thank you did the same, getting the below message. Connection is forwarded from istio-proxy to istio-egress for the TCP port 22 but from the egressgateway can't see much movement.

istio-proxy container logs

[2024-06-18T18:20:56.002Z] "- - -" 0 - - - "-" 0 0 1 - "-" "-" "-" "-" "xx.xx.xx.xx:22" outbound|22||tcp-egressgateway.istio-system.svc.cluster.local xx.xx.xx.xx:35254 xx.xx.xx.xx:22 xx.xx.xx.xx:50056 - - [2024-06-18T18:20:56.002Z] "- - -" 0 - - - "-" 12 0 1 - "-" "-" "-" "-" "xx.xx.xx.xx:22" outbound|22||tcp-egressgateway.istio-system.svc.cluster.local xx.xx.xx.xx:35262 xx.xx.xx.xx:22 xx.xx.xx.xx:50064 - - [2024-06-18T18:21:03.006Z] "- - -" 0 - - - "-" 0 0 2 - "-" "-" "-" "-" "xx.xx.xx.xx:22" outbound|22||tcp-egressgateway.istio-system.svc.cluster.local xx.xx.xx.xx:38172 xx.xx.xx.xx:22 xx.xx.xx.xx:60860 - - [2024-06-18T18:21:03.006Z] "- - -" 0 - - - "-" 12 0 1 - "-" "-" "-" "-" "xx.xx.xx.xx:22" outbound|22||tcp-egressgateway.istio-system.svc.cluster.local xx.xx.xx.xx:38180 xx.xx.xx.xx:22 xx.xx.xx.xx:60868 - -

istio-egressgateway container logs ( TCP port 22)

[2024-06-18T18:21:31.021Z] "- - -" 0 - - - "-" 0 0 0 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_client_address/ xx.xx.xx.xx:22 xx.xx.xx.xx:59576 - - [2024-06-18T18:21:31.021Z] "- - -" 0 - - - "-" 12 0 0 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_client_address/ xx.xx.xx.xx:22 xx.xx.xx.xx:59588 - - [2024-06-18T18:21:38.024Z] "- - -" 0 - - - "-" 0 0 0 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_client_address/ xx.xx.xx.xx:22 xx.xx.xx.xx:59594 - -

istio-egressgateway other pod logs for wildcard entires shows successful connection ( port 443)

[2024-06-18T18:33:04.992Z] "- - -" 0 - - - "-" 171535 47318 87455 - "-" "-" "-" "-" "xx.xx.xx.xx:443" dynamic_forward_proxy_cluster xx.xx.xx.xx:45380 envoy://sni_listener/ envoy://internal_client_address/ xx.xx.com - [2024-06-18T18:33:04.992Z] "- - -" 0 - - - "-" 172849 48203 87455 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_clientaddress/ xx.xx.xx.xx:8443 xx.xx.xx.xx:57128 outbound.443.wildcard.egressgateway.istio-egress.svc.cluster.local -

---
apiVersion: v1
kind: Service
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  type: ClusterIP
  selector:
    istio: egressgateway
  ports:
  - port: 22
    name: tcp22
    targetPort: 22
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: tcp22
  namespace: istio-system
spec:
  hosts:
  - ssh-tcp22.tcp.svc
  addresses:
  - "10.0.0.0/8"
  ports:
  - number: 22
    name: tcp22-port
    protocol: TCP
  resolution: NONE
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 22
      name: tcp
      protocol: TCP
    hosts:
    - ssh-tcp22.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-ssh-tcp22
  namespace: istio-system
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: ssh-tcp22
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ssh-tcp22
  namespace: istio-system
spec:
  host: ssh-tcp22.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-ssh-tcp22-through-egress-gateway
  namespace: istio-system
spec:
  hosts:
  - ssh-tcp22.tcp.svc
  gateways:
  - mesh
  - istio-egressgateway
  tcp:
  - match:
    - gateways:
      - mesh
      destinationSubnets:
      - 10.0.0.0/8
      port: 22
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: ssh-tcp22
        port:
          number: 22
  - match:
    - gateways:
      - istio-egressgateway
      port: 22
    route:
    - destination:
        host: ssh-tcp22.tcp.svc
        port:
          number: 22
      weight: 100

bhat39a commented 3 months ago

@howardjohn any suggestions please.

bhat39a commented 3 weeks ago

Any update on this @howardjohn

bhat39a commented 3 weeks ago

closing this as i have raised a new issue

istio / istio