Expose SSH with a LoadBalancer Service

[bribroder]

Hi there, I've successfully deployed the SSH service with a ClusterIP and connected to it with port-forwarding into the cluster and it works wonderfully--thanks for creating this utility!

I'd like to directly expose the SSH service with a LoadBalancer on AWS but I can't figure out how to get the load balancer to consider the instances "healthy". Right now, they never enter the InService state--so I can establish a connection, but it immediately disconnects. It seems like the load balancer's health check is unable to establish a TCP connection to the SSH service, maybe because of mismatched ports somewhere?

Configuration:

hubUrl: https://jupyter.domain.com

ssh:
  enabled: true
  service:
    port: 22
    type: LoadBalancer

sftp:
  enabled: false

My service shows up with what seems like matching port numbers:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: jupyterhub-ssh
    meta.helm.sh/release-namespace: jupyterhub-dev
  creationTimestamp: "2022-04-02T01:28:27Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: ssh
    app.kubernetes.io/instance: jupyterhub-ssh
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jupyterhub-ssh
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: jupyterhub-ssh-0.0.1-n142.h402a3d6
  name: jupyterhub-ssh
  namespace: jupyterhub-dev
  resourceVersion: "382428194"
  uid: c70f0af0-9683-459f-8c0e-8337a806e6c8
spec:
  clusterIP: 10.100.248.221
  clusterIPs:
  - 10.100.248.221
  externalTrafficPolicy: Cluster
  ports:
  - name: ssh
    nodePort: 32767
    port: 22
    protocol: TCP
    targetPort: ssh
  selector:
    app.kubernetes.io/component: ssh
    app.kubernetes.io/instance: jupyterhub-ssh
    app.kubernetes.io/name: jupyterhub-ssh
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: a1b2c3d4e5f6-123456abcdef.us-east-1.elb.amazonaws.com

The actual LoadBalancer has the same ports and protocols too:

But the instances all report "OutOfService" / "Unhealthy" and when I try to SSH in via this load balancer I am immediately disconnected:

$ ssh -vvvvv -p 22 username@a1b2c3d4e5f6-123456abcdef.us-east-1.elb.amazonaws.com
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /home/ec2-user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to a1b2c3d4e5f6-123456abcdef.us-east-1.elb.amazonaws.com port 22.
debug1: Connection established.
debug1: Local version string SSH-2.0-OpenSSH_8.1
kex_exchange_identification: Connection closed by remote host

I've also tried launching a Network Load Balancer by installing with some service annotations:

  serviceAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

Which creates an otherwise identical service to the above but with an NLB--sadly, I just get a timeout without any success.

Note: my jupyterhub is deployed using the official helm chart but is configured to use a ClusterIP + Ingress and is not exposed to the internet--so I don't use the autohttps/letsencrypt functionality of the hub proxy:

# JupyterHub helm chart proxy config section

proxy:
  service:
    type: ClusterIP

Is there something I can add to the jupyterhub-ssh configuration which will get it working with a LoadBalancer?

Thank you!

[bribroder]

I eventually tracked this down to the NetworkPolicy; I think the only solution for a LoadBalancer is to add a rule permitting traffic by IP range:

  networkPolicy:
    ingress:
      - from:
        - ipBlock:
            cidr: "0.0.0.0/0"
            except: ["10.0.0.0/8"]