search

det-lab / jupyterhub-deploy-kubernetes-jetstream

CDMS JupyterHub deployment on XSEDE Jetstream
0 stars 1 forks source link

NFS sharing issue #10

Closed zonca closed 4 years ago

zonca commented 4 years ago

found issue with NFS sharing, it works fine if the Jupyter Notebook pod is in the same node, but it doesn't work across nodes, which is strange because Kubernetes networking should automatically handle that.

To debug I deployed 2 pods, one on master and one on minion, then 

telnet 10.254.77.77 111

it can connect on master (where the CVMFS/NFS pod is located), hangs on the other node.

zonca commented 4 years ago

this is fixed doing:

sudo iptables --flush
sudo iptables -tnat --flush
sudo systemctl restart docker

from: https://github.com/rancher/rancher/issues/6139#issuecomment-251558161

Now will go ahead with this for now then try to find a better solution.

zonca commented 4 years ago

test this again during the next re-deployment

zonca commented 4 years ago

same issue, it looks like there is a networking issue between the worker and the master node. I tried the fix mentioned above here but it didn't work.

The Jetstream team is working on upgrading the Kubernetes deployment functionality. So I want to wait for them to upgrade. And then directly work on the new setup. So for now my plan is to only run on the master node.

zonca commented 4 years ago

I also reduced resources per user to 3GB and 1 CPU (just for now) Please let me know if anything is still not working.

zonca commented 4 years ago

ok, for now this works fine in new deployment #19, closing for now, reopen if the problem reappears