Python demo broken

[PythonLinks]

I clicked on the demo, the python example is missing an image. Click on it, and I get an error.

The larger problem is that I am not sure what this is.

I think running jupyter notebooks in the browser is hugely important, a great use of Pyodide, but why do we need Jupyter Labs.

What functionality does this not support.

I am just a bit lost.

Chris

[bollwyvl]

Thanks for looking at jyve!

It is a continual work-in-progress: PRs welcome! The core is a reimplementation of most of the stuff that lab needs to do to support interactive computing (e.g kernel messages) without a server. Because it's made of lab, it gains all of the built-in features like media viewers, syntax highlighting and of course the notebook and console UIs and file formats.

The demo is built by scraping the live state of a running lab with jyve installed, with a few extra patches to mimic all the APIs.

The broken link from the demo (being served from GitHub pages, as you may note) is an artifact of this (could easily be hacked away) but I left it as a reminder that this isn't real jupyterlab: of course when installed locally, you can use (or even control from jyve kernels) ipython (or any other kernel) notebooks.

The demo does ship five other kernels though! Brython (a hand-made js implementation) does work in jyve for a subset of python... Check out the notebooks that ship in the demo!

While the kernels are what I really wanted, the static hosting aspect will likely be useful before then. The place I'm currently tinkering is voila:

https://github.com/QuantStack/voila

Basically, instead of serving a full lab to one user with full code execution, a voila/jyve would serve a prepared, customized lab work space to many users. Every user would still get one server kernel per notebook (or other kernel backed UI) but most other features would be disabled, hidden, or replaced so that the builds and ux would be lighter... But all the key code paths would be exactly the same, rather than trying to make nbconvert do it for you.

Anyhow, back to "just" jyve:

A Pyodide kernel is an aspirational goal for jyve, but the latter didn't really have easy-to-integrate releases when 0.35 came out. When JupyterLab 1.0 stable comes out, I'll be doing some work to bring it up-to-date anyway, and will give pyodide a shot again. There are a few other wasm-based language implementations, such as fengari for Lua, which are also very exciting.

The kernels that do exist are all missing some of the nice pieces of interactive computing enabled by the jupyter message spec, in rough order of utility:

• completing
• inspecting
• widgets

These would have to be implemented one by one, even if some common plumbing can be reused. The closest is probably the typescript kernel, as it actually already contains a full LSP server, so it's really just working up the reply messages (it's already getting the request from the "front end").

For widgets, they'll probably just be... The js widgets!

So how can you use this? No idea! But it, and jupyter and pyodide are all open source, so you can use, fork or ignore them any way you like!

On Tue, May 14, 2019, 18:31 PythonLinks notifications@github.com wrote:

I clicked on the demo, the python example is missing an image. Click on it, and I get an error.

The larger problem is that I am not sure what this is.

I think running jupyter notebooks in the browser is hugely important, a great use of Pyodide, but why do we need Jupyter Labs.

What functionality does this not support.

I am just a bit lost.

Chris

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/deathbeds/jyve/issues/42?email_source=notifications&email_token=AAALCRE7QL5EDCGAWW6CAILPVM4T7A5CNFSM4HM6FPIKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTZH65Q, or mute the thread https://github.com/notifications/unsubscribe-auth/AAALCRGZL47EQUSOOFVPHPDPVM4T7ANCNFSM4HM6FPIA .

[PythonLinks]

All very interesting.

Thank you for the excellent tech support. I am quite new to Jupyter, so I need to read more about the architecture, but it seems hugely obvious to me that a kernel running in the client could save a lot of money for anyone who has to pay for their compute servers. And especially for schools, or citizen journalist data analysis which they want to share on the internet.

I will be teaching a course at PyCon Lithuania about Pyodide. People will ask me about Jupyter integration. I will mention this project. By then I will have a better understanding of the advantages of Jupyter Lab over Jupyter Notebooks.

I think that long term, I am likely to run a sprint on Jyve at the next Pycon I attend. Will you be attending any European Pycons?

[SimonBiggs]

A Pyodide kernel is an aspirational goal for jyve, but the latter didn't really have easy-to-integrate releases when 0.35 came out.

Over at https://github.com/pymedphys/pymedphys/pull/252 I have been working on pyodide integration within our docs. With the aim to allow users to press a button which triggers the pyodide download, then have interactive examples and apps right there in the documentation.

To achieve this I have hosted am instance of pyodide at https://pyodide.pymedphys.com/pyodide.js which allows things like the following to be possible:

<html>

<head>
  <script type="text/javascript" src="https://pyodide.pymedphys.com/pyodide.js"></script>
  <script type="text/javascript">
    pythonCode = `
def something_awesome(*args):
    import pydicom

    dataset = pydicom.Dataset()
    dataset.PatientName = 'MacDonald^George'
    print(dataset)

import micropip
micropip.install('pydicom').then(something_awesome)
`

    languagePluginLoader.then(() => {
      return pyodide.loadPackage(['micropip'])
    }).then(() => {
      console.log(pyodide.runPython(pythonCode));
    })
  </script>

</head>

<body></body>

</html>

Just copy the above into a HTML file and it should work...

Note there, which is awesome, that pydicom package is being downloaded from pypi!

There are still some issues outstanding with the pypi implementation (https://github.com/iodide-project/pyodide/issues/413), but it is very cool nonetheless

[PythonLinks]

So first of all it looks like the Python kernel now works. Sure Micropip has problems , but the 37 data science libraries all work.

WOW!

But I am still a bit lost.

Because it's made of lab, it gains all of the built-in features like media viewers, syntax highlighting and of course the notebook and console UIs and file formats.

I thought the only difference between Jupyter Notebook and Jupyter Labs is that the later supported multiple users. Are you telling me that Jupyter lab offers more than Jupyter notebook? Or am I misreading your statement?

The demo is built by scraping the live state of a running lab with jyve installed, with a few extra patches to mimic all the APIs.

So this tells me that you do not need Jupiter lab. I am confused. Maybe you only need Jupyter Lab to run Python kernels?

I see so you have all of those kernels running in the browser, just not python. Is that correct? pretty awesome if you ask me. And now Simon Biggs has got the Python Kernel working.

Why does Viola need nbconvert, if it has a full jupyter (labs?) server?

I am teaching a class on Pyodide on March 25 or 26 at PyCon Lithunia. Would you be avao;ab;e remotely to answer questions? Maybe even to give a lightening talk.

Maybe someone from Pyodide will join the class remotely as well.

Simon, any chance that you will issue a pull request with your patches?

[SimonBiggs]

@PythonLinks here is where I got up to last time:

https://github.com/deathbeds/jyve/issues/33

I certainly could give it another crack...

[PythonLinks]

So, I have been looking at the code. This page looks like it has all of the different kernels.

https://github.com/deathbeds/jyve/tree/master/packages Is that correct?

But there is no Python Kernel.
Is that also correct?

Which would explain why Python is not working. Is that correct?

Here is the cloc command for the js kernel. github.com/AlDanial/cloc v 1.70 T=0.06 s (108.3 files/s, 4856.9 lines/s)

Language files blank comment code

TypeScript 2 23 4 114 JSON 2 0 0 93 Markdown 1 9 0 17 CSS 1 1 0 8

SUM: 6 33 4 232

so 114 lines of typescript gets me a kernel.

So would a Python kernel also be about that long? Maybe worse because there are type conversion issues.

Guidance?

@SimonBiggs it sounds like you have a Python kernel written, but it just does not load numpy. For me that would be a huge step forward. If you could just post that, it would be a great starting point. Naybe I, or someone else, could fix the numpy problem. At least I should be able to create a bug report, either here, or on Pyodide.

At the very least we could do a code review of the existing Python Kernel. "Many eyes make light work. " Perhaps we would spot the problem.

Chris

[SimonBiggs]

See the following scripedforms branch:

https://github.com/SimonBiggs/jyve

[PythonLinks]

That was fast.

Once again I am quite lost. Rather than reading the code, maybe the first thing is to read the docs. I think what I have to do is to read about the Jupyter Kernels and find out what messages there are. Does the Jyve kernel differ from the Jupyter kernels? Maybe I should ask if the message protocol is at all different?

Thanks Chris

[SimonBiggs]

Maybe run a diff between my fork and the original branch and you can see everything that I changed... That might help you know where to look.

On Fri., 17 May 2019, 7:39 am PythonLinks, notifications@github.com wrote:

That was fast.

Once again I am quite lost. Rather than reading the code, maybe the first thing is to read the docs. I think what I have to do is to read about the Jupyter Kernels and find out what messages there are. Does the Jyve kernel differ from the Jupyter kernels? Maybe I should ask if the message protocol is at all different?

Thanks Chris

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deathbeds/jyve/issues/42?email_source=notifications&email_token=ABSBK67XILSPZNH4AODAEV3PVXIAPA5CNFSM4HM6FPIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVTEHOI#issuecomment-493241273, or mute the thread https://github.com/notifications/unsubscribe-auth/ABSBK647EKEREUTAZZX7JTDPVXIAPANCNFSM4HM6FPIA .

[PythonLinks]

Great suggestion.

Thank you for the much needed guidance.

[PythonLinks]

So there is really good documentation on the messaging api in Jupyter. https://jupyter-client.readthedocs.io/en/stable/messaging.html

But there is a bunch of other stuff in your code. Creating a new server. And some other stuff. Is there any other introductory material I should read to understand what is going on? Maybe I should read the tests first?

Maybe I should also read about typescript. I am new to it as well. Any other advice?

[SimonBiggs]

Not sure. I'm too far removed from this at the moment really.

[PythonLinks]

Understood.

I think what you have done so far is awesome.

It is all just a bit scary to me. I am new to Jupyter. I am new to Typescript, I am not that hard core a Javascript developer. So having a framework is hugely helpful.

I will just keep poking at it.

I just think that the ability to run Jupyter Kernels in the browser, and eliminate overloaded compute servers is a hot area.

[SimonBiggs]

I agree, it will be amazing. Especially when Python Jupyter widgets can run without a server... That will be awesome...

[PythonLinks]

https://github.com/iodide-project/pyodide/issues/431

[bollwyvl]

As I mentioned earlier, the core of jyve is in fact a partial implementation of the jupyter message spec, glad you took a look at the upstream docs. Actually, reading the code of ipykernel might help more than anything... really the high road would be to get ipython working directly in pyodide, as that's a couple nerdyears I wouldn't need to spend to get to feature parity.

As it is, on the request side "to the server", I've implemented execute, kernel info, but not inspect and complete, and probably a couple others. On the reply side "from the kernel", we also have execute, display, kernel info, status.

The SLOC is mostly irrelevant, as aside from the boilerplate and message handing, 99% of jyve's total downloaded size is (lazy-loaded when possible) dependencies. Some of the future kernel dependencies, Pyodide and fengari, use wasm, which presents unique problems for including in a jupyterlab build. Here's some documentation about building JupyterLab extensions... while you don't have to use typescript, I can highly recommend it.

There is a python jyve kernel, namely brython. Its python is just not as complete as pyodide. Please look at the example notebooks, or even the brython docs... probably only the turtle demo doesn't work, because they don't package it for some reason. Or something.

Both JupyterLab and pyodide are rapidly evolving. I'm totally fine with starting a jyve 1.x branch which targets the pre-release JupyterLab and accept PRs to that, and becomes master thereafter... But I don't intend to spend any effort on jyve directly (see the voila comments) until JupyterLab 1.0 has reached a stable release. I'd then do a release of the existing kernels, and start (again again) on integrating pyodide (at least). I welcome you to give it a shot (either against 0.35 or 1.0a3)... but I wouldn't probably pick pyoidde-in-jyve as my first full stack web project... it's just a bit complicated at the moment.

[bollwyvl]

Also, regarding widgets without a server: have a gander over at wxyz (demo). While all of these do have a server implementation, they would work without it... if i either had a jyve WidgetManager implementation, or bothered to make them work with notebook classic or nbconvert (which I'm not super excited about, because of the level of shimming that would be required for some of them.... probably need to split up more).

Unlike pyoidide-in-jyve, building a new widget might indeed be a good place to get started on mixed py/ts Jupyter stuff, and will work between Lab 0.35 and 1.0 (it hooks into PhosphorJS, the underlying application framework in JupyterLab) so you don't have to worry about it.

[PythonLinks]

Thanks for the excellent guidance. I think IPython on Pyodide is an awesome place to start.
I have long wanted to start using it. Even if one just did a shell, that would be a big step forward. Even without Jupyter, IPython on WASM is very interesting. I also really really like the idea of starting with the Brython server. @SimonBiggs did you start with Brython?

Too many options.

Lots more stuff to learn about.

[SimonBiggs]

I started with @mdboom's fork of all of @bollwyvl's work. Honestly I just came in and patched a few pieces together, most of what was already there. Anyway at some point I might jump back in. But I probably can't add anything more to this conversation at this point in time.