Unable to get websockets (wss) working

[day0ops]

There isnt much documentation around getting Websockets to work from an injected sidecar. I came across this working example https://github.com/istio/istio/tree/master/samples/websockets but the same rules don't work in my case.

I have a Slack bot that uses websockets using Spring's WebSocketConnectionManager however I keep getting,

2018-10-05 01:41:12.178 ERROR [customer-invitation-service,,,] 1 --- [cTaskExecutor-1] o.s.w.s.c.WebSocketConnectionManager     : Failed to connect

java.nio.channels.ClosedChannelException: null
    at io.undertow.protocols.ssl.SslConduit.write(SslConduit.java:369)
    at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
    at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385)
    at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
    at io.undertow.protocols.ssl.SslConduit$SslWriteReadyHandler.writeReady(SslConduit.java:1260)
    at io.undertow.protocols.ssl.SslConduit$3.run(SslConduit.java:275)
    at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:582)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:466)

Works flawlessly without the sidecar.

My rules are given below,

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: app-gateway
spec:
  # Using Istio default ingress gateway implementation
  selector:
    istio: ingressgateway
  servers:
  - port:
      name: http
      number: 80
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: app-test-virtual-service
spec:
  hosts:
  - "*"
  gateways:
  - app-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: app-test
        port:
          number: 8080
      weight: 100
    websocketUpgrade: true

[day0ops]

As i forgot to mention the websocket address its wss://cerberus-xxxx.lb.slack-msgs.com/websocket/...

[day0ops]

/cc @ymesika Im cc'n you since you worked on the websocket example and PRs. It will be nice to have some documentation around websockets specially around traffic management. I bet I can't be the only person trying to tackle the websockets in the Istio world.

I have fixed this specific issue by adding the following set of rules. My use case was ofcourse completely different to the example as the websocket upgrade DOES NOT initiate from an ingress.

I have mTLS enabled in my mesh. Hence the following.

cat <<EOF | kubectl apply -n apps -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: slack-serviceentry
spec:
  hosts:
  - cerberus-xxxx.lb.slack-msgs.com
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: slack-gateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      name: https
      number: 443
      protocol: HTTPS
    hosts:
    - cerberus-xxxx.lb.slack-msgs.com
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: slack-destination-rule
spec:
  host: is-egressgateway.control.svc.cluster.local
  subsets:
  - name: slack
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: ISTIO_MUTUAL
          sni: cerberus-xxxx.lb.slack-msgs.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: slack-virtual-service
spec:
  hosts:
  - cerberus-xxxx.lb.slack-msgs.com
  gateways:
  - slack-gateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 443
    route:
    - destination:
        host: is-egressgateway.control.svc.cluster.local
        subset: slack
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - slack-gateway
      port: 443
    route:
    - destination:
        host: cerberus-xxxx.lb.slack-msgs.com
        port:
          number: 443
      weight: 100
    websocketUpgrade: true
EOF

[ymesika]

Thanks @nixgadget for documenting this. I will add this information to the sample readme.

[markmarijnissen]

What is the solution?

I cannot get WSS to work with ISTIO

I've tried adding websocketUpgrade: true to the virtualservice, but this does not seem to be sufficient for wss. It does work for ws, though.

[abhyuditjain]

@markmarijnissen Did you find the solution?

[markmarijnissen]

No, I stopped using istio and am using Traefik instead. (As I needed an edge router more than I needed to control the entire service mesh)