Can't connect to Streamlit server

[juliensimon]

This could be related to https://github.com/awslabs/sagemaker-explaining-credit-decisions/issues/5.

When "fixing" get_notebook_name() like so:

def get_notebook_name():
    #with open('/opt/ml/metadata/resource-metadata.json') as openfile:
    #    data = json.load(openfile)
    #notebook_name = data['SolutionPrefix']
    #return notebook_name
    return 'sm-soln-explaining-credit-decisions-notebook'

The notebook generates a proper URL for the streamlit server:

https://sm-soln-explaining-credit-decisions-notebook.notebook.us-east-2.sagemaker.aws/proxy/8501/

I can successfully open that URL, but I get stuck on a "Please wait message". No dashboard is actually displayed. I tried several browsers, deactivated browser plugins, etc.

I can see Streamlit running on the notebook instance:

top - 10:39:31 up  1:04,  0 users,  load average: 0.16, 0.11, 0.14
Tasks: 107 total,   1 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.0%us,  0.5%sy,  0.0%ni, 95.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.5%st
Mem:   3980096k total,  3788248k used,   191848k free,  1384196k buffers
Swap:        0k total,        0k used,        0k free,   654160k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20161 ec2-user  20   0 1347m 242m  71m S  4.7  6.2   0:20.70 streamlit
 9317 ec2-user  20   0  915m 121m  33m S  2.0  3.1   0:19.51 python
 5287 ec2-user  20   0 1320m 114m  24m S  0.7  2.9   0:21.50 jupyter-noteboo
 4679 root      20   0  742m  43m  29m S  0.3  1.1   0:08.29 amazon-cloudwat
 4696 root      20   0  742m  42m  29m S  0.3  1.1   0:07.74 amazon-cloudwat

[thomelane]

Sounds like a jupyter-server-proxy issue. It should get upgraded during OnStart lifecycle config to a version that support websockets. Without web socket support, you're likely to get that 'please wait message forever'. I'll try and reproduce.

[jaypanda]

Check version of jupyterlab by running pip list | grep jupyter jupyter-server-proxy requires jupyterlab>=2. If your notebook instance is running an older version, you may be facing errors. To fix this, you need to update and re-start jupyter-server

pip install -U jupyterlab
sudo initctl restart jupyter-server --no-wait

Hope this helps

[HwangJohn]

@jaypanda When I type the commands, jupyter notebook can't start again.

[jaypanda]

@jaypanda When I type the commands, jupyter notebook can't start again.

Try launching the notebook instance from AWS console, typically the url */tree should be loading correctly and linux terminals as */terminals/1

[edisongustavo]

Hello, I've found a way to solve this issue. The problem is that SageMaker Notebooks come with a very old nbserverproxy version (0.3.2), which is from 2017 according to PyPI.

The project has since then been renamed to jupyter-server-proxy. The solution is to remove the old nbserverproxy package and install the new jupyter-server-proxy package.

You can do so by running these commands. They can be run in a "Terminal" or in a OnStart - Lifecycle config:

source /home/ec2-user/anaconda3/bin/activate JupyterSystemEnv
pip uninstall --yes nbserverproxy
pip install --upgrade jupyter-server-proxy
initctl restart jupyter-server --no-wait

By doing this I am able to access the Streamlit server on: https://<notebook-name>/proxy/8501/

[sam-cohan]

Unfortunately the recommended solution does not work. I have

➜  dl-team git:(streamlit_app) ✗ pip show jupyterlab
Name: jupyterlab
Version: 3.0.9
Summary: The JupyterLab server extension.
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /home/ec2-user/SageMaker/.pyenv/versions/3.7.10/envs/pydev37/lib/python3.7/site-packages
Requires: jupyter-packaging, nbclassic, jinja2, tornado, jupyter-server, ipython, packaging, jupyter-core, jupyterlab-server
Required-by: 

➜  dl-team git:(streamlit_app) ✗ pip show jupyter-server-proxy
Name: jupyter-server-proxy
Version: 3.2.2
Summary: Jupyter server extension to supervise and proxy web services
Home-page: https://github.com/jupyterhub/jupyter-server-proxy
Author: Ryan Lovett & Yuvi Panda
Author-email: rylo@berkeley.edu
License: BSD-3-Clause
Location: /home/ec2-user/SageMaker/.pyenv/versions/3.7.10/envs/pydev37/lib/python3.7/site-packages
Requires: aiohttp, jupyter-server, simpervisor
Required-by: jupyterlab-commenting-service

➜  dl-team git:(streamlit_app) ✗ pip show nbserverproxy
WARNING: Package(s) not found: nbserverproxy

But still getting on please wait screen forever. Interestingly streamlit==1.11.0 works but does not have the web modules. Is this something you can look into again for the newer releases?

[absynthe]

I have the same issues, the app is sucessffuly deployed, but I get socket timeout errors in the console. Streamlit works just fine in SageMaker Studio, but not in Notebooks.

Anyone found a solution to this?

[VadneyK]

If you're getting the follwing error:

 (JupyterSystemEnv) sh-4.2$ initctl restart jupyter-server --no-wait
sh: initctl: command not found

Do this instead

sudo systemctl restart jupyter-server

initctl is for AWS Linux 1, systemclt for AWS Linux 2

Additonally I followed the suggested solution but still am not able to connect. There was a similar suggested solution stating the issue is that streamlit is in the localhost of the sagemaker server and not my local computer so I can't access it unless I expose the port. I tried this but it's not working either ;(