Support for Deepnote

[ochen1]

Is your feature request related to a problem? Please describe. Deepnote just launched a few days ago, and it looks like they have 24/7 instances. I feel many people would love to be able to SSH & VNC into their 24/7 Deepnote servers.

Describe the solution you'd like I tried to run remocolab on Deepnote, but nvida-smi not being present caused some errors.

Following the StackOverflow answer here, I was able to at least get the python-apt module working.

Would support for Deepnote be possible?

[demotomohiro]

I tried to run remocolab on deepnote but I cannot login to ssh.

I ran following code on the notebook cell:

!sudo apt update && sudo apt upgrade -q -y
!sudo apt install -q -y -o=Dpkg::Use-Pty=0 python3-apt
!pip3 install git+https://github.com/demotomohiro/remocolab.git@add-deepnote

with open("remocolab_run.py", "w") as f:
  f.write("""
import sys, os, tempfile
sys.path.append("/usr/lib/python3/dist-packages")
sys.path.append("/opt/venv/lib/python3.7/site-packages")

with open("/etc/ssh/sshd_config", "a") as f:
  f.write("\\n\\nListenAddress localhost:58387")

import remocolab
os.chdir(tempfile.mkdtemp(prefix = "remocolab"))
#remocolab.setupSSHD(check_gpu_available = False, tunnel = "argotunnel")
remocolab.setupVNC(check_gpu_available = False, tunnel = "argotunnel")

Then ran following command on deepnote's terminal:

sudo python3 remocolab_run.py

sshd worked on the server and setupVNC ran without errors but when I tried to connect the server, I got following error:

Connection closed by UNKNOWN port 65535

Note:

• apt update need to be executed before installing python3-apt.
• Unlike Google colab, deepnote runs cells as non-root user. So I need to use sudo when I install packages.
• When command line program wait for user input, deepnote cells cannot take user input and wait for forever.
  • That means I have to run remocolab from deepnote's terminal
• Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.
• sshd doesn't run in default config. It need to listen port other than 22.

It seems deepnote shutdown the instance if I didn't use it for long time like google colab. I run python3 -c "import time;time.sleep(1111111111)" in deepnote terminal and closed the tab. When I open deepnote again after few minutes, that command was running. But when I reopen the tab after about 1 hour, that command was not running.

[ochen1]

Hi! Any updates on Deepnote support?

I noticed a new branch being created, which fortuately fixes the nvidia-smi FileNotFound exception.

I found out how to install remocolab and its dependencies onto Deepnote, but ngrok is giving me

"Connection refused" (on ssh client when trying to connect)

and

"There are no tunnels currently online in your account." (on ngrok dashboard)

(at the time of testing _(now))

argotunnel sorta works, but not really.

INFO[2020-12-23T01:07:37Z] Route propagating, it may take up to 1 minute for your new route to become functional
ERROR[2020-12-23T01:07:49Z] Cannot connect to remote: dial tcp 127.0.0.1:58387: connect: connection refused
ERROR[2020-12-23T01:07:49Z] CF-RAY: 605e30156af01bf4-SEA Proxying to ingress 0 error: websocket: bad handshake

It gets me closer than ever before to a connection, but at the last minute the above error message appears on the server.

[ochen1]

Hello! It's odd, but it appears my findings are inconsistent with your notes!

Notes: ^{(by "Deepnote" I may mean "Deepnote's default docker instance")}

• Deepnote provides a handy Terminal, which I prefer using over the notebook
  • The terminal allows spawning new interactive Python REPL processes
  • Programs like htop work and are great for debug
• The default version of Python on Deepnote is Python 3.7.9
  • Installing any other version of Python is very complicated (touch docker, get into a dependency hell with apt)
• Deepnote does not have apt cache updated by default, need to run apt update before anything
• python(3)-apt does not install successfully on Deepnote (pip and apt both fail installation)
  • pip fails due to ModuleNotFound: DistUtilsExtra error
    • If the notebook cell is not running as root (as per your speculations), then pip would essentially be installing to (3) different places, which might cause confusion and/or errors: /root/.local/lib/python3.7/ (does not exist on a fresh new instance), /home/jovyan/.local/lib/python3.7/ (does not exist on a fresh new instance), and /usr/lib/python3.7/
  • the package manager apt succeeds when installing python-apt
    • apt still can't be imported, raising a ModuleNotFound exception at import
    • apt_pkg can't be imported either
    • the import names above were tested (ie. raised a ModuleNotFound exception) in the folowing situations:
      1. restarting the notebook (NOT the instance), in a notebook cell
      2. using a new instance of the Python 3 REPL
  • It appears usage of the python-apt module is still possible though, this StackOverflow question has some useful answers. Also see my (semi-)working code block below.
• Deepnote runs cells as the user root. The terminal is also running as root.
• Deepnote has no support for stdin, nor support for ipywidgets (yet) (so must use terminal to run remocolab.py _{(for now)})
• The content inside Deepnote's default ~/work directory is persistent across runtimes / sessions / instances / reboots
• The $HOME for user root in the bash shell is /home/jovyan, but this is inconsistent with /etc/passwd
  • (my speculations are that) this may be due to some line overriding the $HOME variable in .bashrc or something similar

Here is the code block I used for testing:

# Install remocolab
%cd ~/
!git clone -b add-deepnote https://github.com/demotomohiro/remocolab.git
%cd remocolab/
!apt -qq update
!apt -qq install -y python3-dev python3-setuptools python3-distutils python3-distutils-extra python3-apt --reinstall
!mv /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.so
!mv /usr/lib/python3/dist-packages/apt_inst.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_inst.so
!echo "pyngrok" >> requirements.txt
!pip -q install -r requirements.txt
!cp -r /usr/lib/python3/dist-packages/* ~/remocolab/
!cd /tmp/ && wget "https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip" && unzip ngrok-stable-linux-amd64.zip && rm ngrok-stable-linux-amd64.zip && mv /tmp/ngrok /usr/bin && chmod +x /usr/bin/ngrok
# !cd ~/remocolab/ && wget "https://github.com/cloudflare/cloudflared/releases/download/2020.11.11/cloudflared-linux-amd64" && mv ./cloudflared-linux-amd64 ./cloudflared && chmod +x ./cloudflared

# Now run the following command in a terminal (so you can access stdin):
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD()"
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD(tunnel='argotunnel')"

Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.

I'm not so sure about this. From my experiments, anything inside the ~/work directory is persistent storage.

It appears that this is the case because ~/work/ is a directory symlink to -> /work/

/work is mounted as a remote filesystem, as seen here:

This makes sense, because the remote persistent filesystem caps free accounts to 5GB of storage. However, the local usable filesystem is much larger, at 100GB. Anything inside the local filesystem will, however, be purged during a hardware restart or instance stop. Note that this means every Python module installed must be reinstalled after a hardware restart (unless a virtualenv is used inside persistent storage), because the directory /usr/share/python3/ is not persistent.

It also appears Deepnote gives 750 hours of free uptime per month, meaning 100% uptime of a machine, so you can have a persistent instance! _{(wouldn't bet on it though, the instance might get powered off)}

I've tested this as my friend is now hosting a Discord bot on Deepnote, without issue (no, this is not against the ToS_{I don't think})

The bot is still online, even after over 8 hours. :smiley:

dashboard still displays online status

session running on standard runtime long after the tab was closed

cell running on standard runtime long after the tab was closed

and the bot is still online

and well and responding to events

[ochen1]

This is the error message I am getting with cloudflared:

[ochen1]

Is this expected? I am getting a <defunct> with ps aux | grep sshd :thinking:

[rayanfer32]

Cloudflare takes 2 or more minutes sometimes to establish connection

On Wed, 23 Dec, 2020, 11:54 pm ochen1, notifications@github.com wrote:

This is the error message I am getting with cloudflared: [image: cloudflared error message screenshot] https://user-images.githubusercontent.com/59662605/103020560-d8d62d00-4505-11eb-920a-fe5ca09912ce.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750419332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5U4PWVIOEXAPM4B4LTSWIYUXANCNFSM4TIHX33A .

[rayanfer32]

Wait for sometime....it will work.. Im also faving the delay to establish connection

On Wed, 23 Dec, 2020, 11:55 pm ochen1, notifications@github.com wrote:

Is this expected? I am getting a with ps aux | grep sshd 🤔

[image: defunct screenshot] https://user-images.githubusercontent.com/59662605/103026555-73883900-4511-11eb-9a0c-1c4854076622.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750419658, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5S5TYK2Z3JNBL6GZITSWIYYBANCNFSM4TIHX33A .

[ochen1]

Nope, nvm, I was just being stupid lol.

For some reason running remocolab failed to start sshd, so I had to invoke it manually via /usr/sbin/sshd.

Also, remocolab did not automatically add port configuration to /etc/ssh/sshd_config for some reason. I added it manually using terminal + vim.

Apparently, the error I posted before (connection refused; bad handshake) was because cloudflared could not find the open port on localhost.

Now I am getting this error:

Connection closed by UNKNOWN port 65535

Looks like @demotomohiro also got that error!

[rayanfer32]

Ohh i see... plz let me know how you get that one fixed... same thing happening for me...within 10 mins of using remocolab

On Thu, 24 Dec, 2020, 12:06 am ochen1, notifications@github.com wrote:

Nope, nvm, I was just being stupid lol.

For some reason running remocolab failed to start sshd, so I had to invoke it manually via /usr/sbin/sshd.

Also, remocolab did not automatically add port configuration to /etc/ssh/sshd_config for some reason. I added it manually using terminal

• vim.

Apparently, the error I posted before (connection refused; bad handshake) was because cloudflared could not find the open port on localhost.

Now I am getting this error:

Connection closed by UNKNOWN port 65535

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750423624, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5QM33PQ2QUPOFS7SELSWI2DLANCNFSM4TIHX33A .

[ochen1]

@Rayanfer32 Are you running on Deepnote? I'm not sure if this issue is related to what you are trying to do.

[ochen1]

It looks like Deepnote is blocking port 22! Even root is unable to start a server on port 22.

Trying to use sshd with port 22 causes <defuct> in ps aux and trying to run SimpleHttpServer or http.server on port 22 causes a PermissionError (even as root). Please see attatched screenshot below:

[rayanfer32]

Its happening in colab too

On Thu, 24 Dec, 2020, 12:18 am ochen1, notifications@github.com wrote:

@Rayanfer32 https://github.com/Rayanfer32 Are you running on Deepnote? I'm not sure if this issue is related to what you are trying to do.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750427203, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5XCPJP5ZXLMF2NHA5LSWI3OHANCNFSM4TIHX33A .

[ochen1]

This issue is for adding support for Deepnote #76

It is note related to Google Colab or anything happening there.

If you are experiencing an issue, feel free to open another new issue for it. Just make sure to look for duplicates :hugs:

[rayanfer32]

Its fine... i didnt notice the title... btw support for tunnels seems to be getting blocked eventually

On Thu, 24 Dec, 2020, 12:26 am ochen1, notifications@github.com wrote:

This issue is for adding support for Deepnote #76 https://github.com/demotomohiro/remocolab/issues/76

It is note related to Google Colab or anything happening there.

If you are experiencing an issue, feel free to open another new issue for it. Just make sure to look for duplicates 🤗

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750429752, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5UVN2FXT4SDJKRHNPTSWI4MLANCNFSM4TIHX33A .

[ochen1]

This is odd: sshd is started on port 8022 without error.

ngrok is forwarding to localhost (which 0.0.0.0 covers) port 8022.

Yet I am unable to connect to ssh:

EDIT: I think I've found some mistakes in the above screenshots:

1) Logging in as a non-existent user: the user colab does not exist on Deepnote. What am I doing! It exists now, as it was created automatically by remocolab.

2) PermitRootLogin needs to be enabled in sshd_config in order to login as root.

But its still not working. (Connection closed)

[rayanfer32]

Can you verify ssh inside notebook using paramiko or some tool

On Thu, 24 Dec, 2020, 12:34 am ochen1, notifications@github.com wrote:

This is odd: sshd is started on port 8022 without error. [image: sshd] https://user-images.githubusercontent.com/59662605/103029123-599d2500-4516-11eb-8f6c-fdf778687003.png

ngrok is forwarding to localhost (which 0.0.0.0 covers) port 8022. [image: ngrok] https://user-images.githubusercontent.com/59662605/103029161-6caff500-4516-11eb-9888-0b965c0a9b3a.png

Yet I am unable to connect to ssh: [image: ssh] https://user-images.githubusercontent.com/59662605/103029356-e0520200-4516-11eb-86b6-47d7b7d7c266.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/demotomohiro/remocolab/issues/76#issuecomment-750432572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI3MT5QQFTOY4GK4H4JVEATSWI5KNANCNFSM4TIHX33A .

[ochen1]

After some more debugs, I got the following error message (found an error at last)!

When running sshd on debug mode with -d flag, I get the following error, and then the program terminates:

root@deepnote:~/work $ /usr/sbin/sshd -d
debug1: sshd version OpenSSH_7.9, OpenSSL 1.1.1d  10 Sep 2019
debug1: private host key #0: ssh-rsa SHA256:BJ2D+KSTupbYM+91QLoOauGSt5yEcQD1yP5PhOSjBsM
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:xUMDrs7kqf6YG+JfxJjSt357LK8sII5HEtF13AjhXvg
debug1: private host key #2: ssh-ed25519 SHA256:UJKQ3LFUbesZpTpgW99f3OQ2bzOb6nfrADuJ4meGccg
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: Set /proc/self/oom_score_adj from 925 to -1000
debug1: Bind to port 9558 on 0.0.0.0.
Server listening on 0.0.0.0 port 9558.
debug1: Bind to port 9558 on ::.
Server listening on :: port 9558.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 47388 on 127.0.0.1 port 9558
debug1: Client protocol version 2.0; client software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
chroot("/run/sshd"): Operation not permitted [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 2072
debug1: audit_event: unhandled event 12

(9558 was a random arbitrary port I chose)

EDIT: You can use sshd -T to check the current sshd server config and ssh -v -v -v -v -o UserKnownHostsFile=/dev/null for client.

[ochen1]

At last! I got a connection (through ngrok)!!! (without using remocolab)

[rayanfer32]

At last! I got a connection (through ngrok)!!! (without using remocolab)

Woah thats great man... Can you plz provide your script for the setup. Thanks

[rayanfer32]

Deepnote allow users to use custom docker images... more over here: https://docs.deepnote.com/environment/custom-environments

This means we can load docker with xfce4 and cloudflared preinstalled

[ochen1]

This means we can load docker with xfce4 and cloudflared preinstalled

The last time I checked, the only docker base image supported by Deepnote the is the default image, provided by Deepnote. :sob:

This means you will still have to take time to install the dependencies. RIP. :slightly_frowning_face:

It might be, however, to create a Dockerfile to install the dependencies for us, which may or may not reduce the startup time.

Either way, Deepnote offers 24/7 machines, whose instances never have to be powered off (unlike the 12h limit of Google Colab), so installation is only necessary once, as long as you keep your "hardware" running, for example with the following code.

import time
while True:
    time.sleep(10000)

[ochen1]

cloudflared

ngrok is not banned (has not been banned yet) on Deepnote, so we can continue to use it. ¯\_(ツ)_/¯

[ochen1]

Update: Deepnote added support for custom Docker images