search

jupyterhub / zero-to-jupyterhub-k8s

Helm Chart & Documentation for deploying JupyterHub on Kubernetes
https://zero-to-jupyterhub.readthedocs.io
Other
1.52k stars 791 forks source link

[Proxy] 503 socket hang up #1665

Closed mhwasil closed 4 years ago

mhwasil commented 4 years ago

Hi,

Jupyterhub keeps loading in the browser without even opening the login page. My environment:

z2jh: 0.9.0-beta.3, 0.9.0 (tried both)
Kubernetes version: v1.16.9, v1.16.8, v1.16.0, v1.15.0 (tried all of them)
Cloud provider: OpenStack
Helm: v2.16.3

I used NodePort as service with specified port. When I opened it using one of the node ip addresses with the port, it just got stuck at loading without even showing login page.

Here is the logs from proxy and hub:


When I did nmap from different machine to the node ip and the port, I got this:

PORT STATE SERVICE VERSION 31011/tcp open unknown 1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service : SF-Port31011-TCP:V=7.01%I=7%D=5/13%Time=5EBBEA3D%P=x86_64-pc-linux-gnu%r(G SF:etRequest,15E,"HTTP/1.1\x20302\x20Found\r\nserver:\x20TornadoServer/6\ SF:.0.4\r\ncontent-type:\x20text/html\r\ndate:\x20Wed,\x2013\x20May\x2020 SF:20\x2012:38:21\x20GMT\r\nx-jupyterhub-version:\x201.1.0\r\naccess-con SF:trol-allow-headers:\x20accept,\x20content-type,\x20authorization\r\ncon SF:tent-security-policy:\x20frame-ancestors\x20'self';\x20report-uri\x20/h SF:ub/security/csp-report\r\nlocation:\x20/hub/\r\ncontent-length:\x200\r\ SF:nconnection:\x20close\r\n\r\n")%r(RTSPRequest,2F,"HTTP/1.1\x20400\x20B SF:ad\x20Request\r\nConnection:\x20close\r\n\r\n")%r(RPCCheck,2F,"HTTP/1. SF:1\x20400\x20Bad\x20Request\r\nConnection:\x20close\r\n\r\n")%r(DNSVersi SF:onBindReq,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r\nConnection:\x20clos SF:e\r\n\r\n")%r(DNSStatusRequest,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r SF:\nConnection:\x20close\r\n\r\n")%r(Help,2F,"HTTP/1.1\x20400\x20Bad\x20 SF:Request\r\nConnection:\x20close\r\n\r\n")%r(SSLSessionReq,2F,"HTTP/1.1 SF:\x20400\x20Bad\x20Request\r\nConnection:\x20close\r\n\r\n")%r(TLSSessio SF:nReq,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r\nConnection:\x20close\r\n SF:\r\n")%r(Kerberos,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r\nConnection: SF:\x20close\r\n\r\n")%r(SMBProgNeg,2F,"HTTP/1.1\x20400\x20Bad\x20Request SF:\r\nConnection:\x20close\r\n\r\n")%r(X11Probe,2F,"HTTP/1.1\x20400\x20B SF:ad\x20Request\r\nConnection:\x20close\r\n\r\n")%r(FourOhFourRequest,181 SF:,"HTTP/1.1\x20302\x20Found\r\nserver:\x20TornadoServer/6.0.4\r\ncont SF:ent-type:\x20text/html\r\ndate:\x20Wed,\x2013\x20May\x202020\x2012:38:2 SF:6\x20GMT\r\nx-jupyterhub-version:\x201.1.0\r\naccess-control-allow-he SF:aders:\x20accept,\x20content-type,\x20authorization\r\ncontent-security SF:-policy:\x20frame-ancestors\x20'self';\x20report-uri\x20/hub/security/c SF:sp-report\r\nlocation:\x20/hub/nice%20ports%2C/Tri%6Eity.txt%2ebak\r\n SF:content-length:\x200\r\nconnection:\x20close\r\n\r\n")%r(LPDString,2F," SF:HTTP/1.1\x20400\x20Bad\x20Request\r\nConnection:\x20close\r\n\r\n")%r( SF:LDAPBindReq,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r\nConnection:\x20cl SF:ose\r\n\r\n")%r(SIPOptions,2F,"HTTP/1.1\x20400\x20Bad\x20Request\r\nCo SF:nnection:\x20close\r\n\r\n");

mhwasil commented 4 years ago

I found the problems and it's apparently not related to z2jh, instead to kubernetes CNI. I install k8s on openstack vms with calico which does not encapsulate the pod ip with the host ip addresses. The problem is openstack has port security that prevents un-allowed ips to communicate with the vm. That is probably why I got socket hang up in the hub container.

Adding kubernetes_service_addresses and kube_pods_subnet to each vm ports solve this problem. This is further explained here.