search

fonsp / PlutoUtils.jl

Nifty scripts to go with Pluto.jl
The Unlicense
22 stars 11 forks source link

JupyterLab integration #4

Open fonsp opened 4 years ago

fonsp commented 4 years ago

Help welcome!

I managed to get this working for one setup, but I am not sure how to make it more general so that other people can install it.

The repository is here:

https://github.com/fonsp/pluto-on-binder

with a demo on binder:

https://mybinder.org/v2/gh/fonsp/pluto-on-binder/master (it's one of the options when launching a new kernel/notebook) or you can skip the Jupyter UI and go straight to Pluto: https://mybinder.org/v2/gh/fonsp/pluto-on-binder/master?urlpath=pluto

I also got it running as jupyterlab extension with a nice button and logo

It sets up jupyter for binder with the http server proxy extension, and installs a mini python package that is used by Jupter as an extension, available through ?urlpath=pluto.

Goals

  1. Instructions for other people who have JupyterHub/Lab (not sure which is which) running on how to add Pluto. Should plutoproxy be a registered Python package?
  2. (down the road) Integrate JupyterLab-specific UI into Pluto, like the RAM usage, a button to go back to the Lab Hub menu thingy, Pluto notebooks as a tab within Lab, instead of a full-screen window, and what else?
shashankp commented 4 years ago

https://github.com/lungben/PlutoUtils.jl/pull/3 is to make Pluto work with Binder, which is essentially Jupyter. Should work in local/binder/any other env

This works right now

docker run -it -p 8080:8080 plutojl/pluto-binder:latest jupyter notebook --ip=0.0.0.0 --port=8080
fonsp commented 3 years ago

We can run Pluto notebooks as Lab tabs: https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html

Also set a very high HTTP timeout (60 works)

fonsp commented 3 years ago

@koehlerson wrote this handy guide!

https://www.maximiliankoehler.de/posts/pluto-server/

koehlerson commented 3 years ago

let me know if you need the post without the teaching stuff overhead somewhere

fonsp commented 3 years ago

Could someone take a look at: https://github.com/illumidesk/jupyter-pluto-proxy Which Pluto version does it use? Will it be stuck on an outdated version? @jgwerner

It's listed in the official jupyter server proxy docs: https://jupyter-server-proxy.readthedocs.io/en/latest/examples.html

koehlerson commented 3 years ago

The repo itself or the binder? Binder used 12.4. when I tried. I think it will work ootb as long as these lines don't change for starting a Pluto.jl Server https://github.com/IllumiDesk/jupyter-pluto-proxy/blob/main/jupyter_pluto_proxy/__init__.py#L23

jgwerner commented 3 years ago

HI @fonsp ! First of all kudos for developing Pluto, 💯 We can update Pluto in this example repo no problem, should we just update Pluto to the latest stable release? We could also add a bot to automatically create a PR to update the version when you cut a new release. In our case, the use-case was to set up a POC since our tool uses the JupyterHub and some folks are more on the Julia side of things than Python.

@koehlerson regarding the default command, we can update that too.

As far as the perceived official status goes, we sometimes send a PR to the jupyter-server-proxy repo and/or add jupyter-server-proxy as a repo topic. Both of these were recommendations from @betatim from this thread. To avoid duplicating "official" repos, we would gladly transfer what we did to your GH account/org and help contribute to the repo in your account.

fonsp commented 3 years ago

@koehlerson @barche @jgwerner @GiggleLiu You have all managed to run Pluto on JupyterLab, that's great! @koehlerson even wrote a detailed guide: https://www.maximiliankoehler.de/posts/pluto-server/

Let's work together to create a simple and clear way for people to install Pluto as an extension.

I think @koehlerson has done most of the work already, but still left is:

And then we can close this issue!

fonsp commented 3 years ago

FYI, the differences between pluto-on-binder and pluto-on-jupyterlab are:

koehlerson commented 3 years ago

Hey Fons, this sounds good. I pushed the setup I use on my server, which should work for the current version of Pluto.jl as well. However, I cannot confirm and I have a lot of deadlines this week. Maybe @GiggleLiu can try it? Sorry again @GiggleLiu for the wrong pip plugin link in the tutorial. I'll fix all links and commands as soon as the new repo is good to go for reproduction.

Feel free to remove the .toml and .yml files/file. I wasn't sure if they are leftovers from the fork or if they are actually needed. IMHO they can be removed, but I'm not confident enough to guarantee it.

If you are interested in the interaction of WGLMakie and such a Pluto.jl server, I can also describe in a short tutorial how admins can set this up.

barche commented 3 years ago

This is a very nice initiative, I'm certainly willing to describe more in detail what our setup is and integrate pluto-on-jupyterlab into our workflow. I had a quick look at @koehlerson 's nice guide, but most of it is actually about setting up jupyterhub (i.e. the multi-user server) itself, and there are many different approaches for that. We use the docker spawner, where each logged in user starts up jupyterlab in his own docker container and can also launch Pluto from there by clicking a button.

The actions needed to add Pluto are actually quite simple and just a short part, so we should maybe delimit what part exactly we want to describe as part of the Pluto documentation.

GiggleLiu commented 3 years ago

Hi, @fonsp and @koehlerson

I just tried this exciting new feature following @koehlerson 's nice blog. From a user perspective, I feel the blog is clearly written. So far, except the user permission issue, it works perfectly. Hope the following feedbacks can be helpful.

about the blog writting

  1. For AWS users, is it nessesary to setup static IP? I just ignored the first section, hope doing this is safe. (aws setup reference: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html)

  2. add mkdir /usr/share/juliapackages before the chown command.

  3. The nodejs of ubuntu source is too old. This guide is helpful to me: https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/

  4. jupyterlab 2.2.8 works, however, the latest (default) 3.0.1 does not.

  5. When I add a new user without the -m option, the server errors on home folder now exist. sudo useradd -aG juliausers studentXX -> sudo useradd -amG juliausers studentXX. This should be double checked. The error I see is

    
[I 2021-01-06 18:24:05.452 JupyterHub spawner:1451] Spawning /opt/jupyterhub/bin/jupyterhub-singleuser --port=34457 --SingleUserNotebookApp.default_url=/lab

Couldn't set CWD to /home/grubby ([Errno 2] No such file or directory: '/home/grubby')

Traceback (most recent call last):

File "/opt/jupyterhub/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get 

value = obj._trait_values[self.name]

KeyError: 'runtime_dir' 



6. It is helpful to tell users how to read the log: `sudo journalctl -u jupyterhub`

## about pluto-jupyter
1. Is there a better way to add a package. Like colab, one can freely add packages and remove it after the session dies.
2. Is it possible to specify the sample folder so that users can see the prepared samples.
3. I have to run `chown` and `chmod 777` on `/usr/share/juliapackages` repeatedly to make sure the permissions are correct. This is the issue that makes me headache.
koehlerson commented 3 years ago

thanks for the input @GiggleLiu !

I think we clearly need to differ use cases. For me I want that no one adds packages except sudo users. Outside the realm of teaching it makes sense to let users add packages on their own. But, if there are multiple ones and you share a common depot path, then you need to be careful. With the default settings, i.e. not setting the common julia depot path (delete every JULIA_DEPOT_PATH variable you've set during the tutorial on my website), packages will be installed under /home/studenXX/.julia/. So, each user has its own package directory and can do with it whatever she/he wants to do. I do not know enough about AWS, but in general the computer where I installed all of this should have the same ip whenever I want to access it. I guess you have some sort of ip or url from aws which stays the same? Or at least the aws system redirects you always to your machine/account allocated ressources

@barche can you access with this approach the proxy pluto server? So, something like this:

user = ENV["USER"]
external_ip = "134.147.244.113/user/$user/proxy/"
port = 3389

Thanks again to everyone contributing to this!

GiggleLiu commented 3 years ago

Thanks for the reply.

With the default settings, i.e. not setting the common julia depot path (delete every JULIA_DEPOT_PATH variable you've set during the tutorial on my website), packages will be installed under /home/studenXX/.julia/. So, each user has its own package directory and can do with it whatever she/he wants to do.

nice!

I do not know enough about AWS, but in general the computer where I installed all of this should have the same ip whenever I want to access it. I guess you have some sort of ip or url from aws which stays the same? Or at least the aws system redirects you always to your machine/account allocated ressources.

Yes, an AWS host has a static IP and also a url. I think this is worth mention in the blog so that people can skip this sector with confidence.

jgwerner commented 3 years ago

@fonsp Let me know if you would like us to hand over the repo to you to ensure proper upgrades. We would also be glad to help you out here and perhaps attempt to automate creating PRs when you release a new version.

My 2 cents is that JupyterLab version 3.0 was just released as far as I know those extensions are not compatible with 1.x/2.x extensions.

fonsp commented 3 years ago

Whatever you think is best! You all understand this much better than I do

jgwerner commented 3 years ago

Whatever you think is best! You all understand this much better than I do

@fonsp That I doubt very much 😄 - I vote for passing this on to your account (or an org that you designate) and then we would be glad to contribute any/all code we feel necessary to the code in that location. What do you think?

jgwerner commented 3 years ago

@fonsp could you create a repo for your account for jupyter-pluto-proxy (or any other name that you would like)? With that, I can send you a PR with the legacy code-base and update the license from BSD-3 Clause (we left it with BSD for the sake of consistency with the Jupyter project) to Unlicensed so that you may add the license of your choosing.

I'll do my best to update deps/docs for the first PR to your account and then open issues in your repo for items that we have taken note of. As a side note, we can provide the step to add the jupyter-pluto-proxy to the Jupyter Server Proxy "official" examples and remove the reference to ours to avoid confusion. We're excited to help you get this started with you!

fonsp commented 3 years ago

Okay! But don't we already have this here? https://github.com/fonsp/pluto-on-jupyterlab/tree/master/plutoserver

It has to be clear for people what they need to install (and how they update Pluto after it is installed), so having a single repository would be helpful.

pankgeorg commented 3 years ago

@GiggleLiu the compatibility with 3.0.1 seems to be a problem with jupyter -- see this comment (They're waiting for extension compatibility before updating Docker stacks to 3.0.1)

jgwerner commented 3 years ago

Okay! But don't we already have this here? https://github.com/fonsp/pluto-on-jupyterlab/tree/master/plutoserver

It has to be clear for people what they need to install (and how they update Pluto after it is installed), so having a single repository would be helpful.

@fonsp Ah cool! I'll review that repo then and send a PR in case we see something that we have on our end that is missing from yours. Then we will archive ours and add yours as a reference in the Readme.

fonsp commented 3 years ago

https://xkcd.com/927/

fonsp commented 3 years ago

@GiggleLiu @jgwerner @koehlerson Can we try to create a single guide that explains how to install pluto as a jupyterlab extension? I still don't know what to say when people ask me :(

barche commented 3 years ago

I'm currently using the unmerged PR from @pankgeorg, in the way described here: https://github.com/barche/plutohub-juliacon2021

jgwerner commented 3 years ago

@fonsp we are just using the jupyter-server-proxy extension and then configure it to run Pluto: https://github.com/IllumiDesk/jupyter-pluto-proxy.

johannes-fischer commented 2 years ago

I have been looking around for a way to share Pluto notebooks with students in a way that they can run them on binder. I have found various github repos and issues in that direction, but I couldn't find something working. Is this still work in progress, or is there a working solution for this somewhere around?

fonsp commented 2 years ago

@johannes-fischer You should try https://github.com/JuliaPluto/static-export-template/ ! Open an issue there if something is not working

johannes-fischer commented 2 years ago

@fonsp thanks, will have a look there!