Support JupyterLab - Githubissues

astrojuanlu commented 4 years ago

Edit: See current status in this comment.

At the moment, czml3.widget.CZMLWidget has a straightforward _repr_html_ that prints some HTML. However, this is not how custom widgets are supposed to be implemented [citation needed]

We should write our own custom Jupyter widget so it's compatible, at least, with modern JupyterLab.

Edit: Adding more information to this description for completeness

Rather than a JupyterLab extension, if I understand correctly we would need to create a custom Jupyter widget. Therefore, we should be using either https://github.com/jupyter-widgets/widget-cookiecutter (JavaScript, barebones) or https://github.com/jupyter-widgets/widget-ts-cookiecutter (TypeScript + bells & whistles), which are described in https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Low%20Level.html

However, we should wait for the TypeScript one to be updated: https://github.com/jupyter-widgets/widget-ts-cookiecutter/pull/83

These cookiecutters already make use of https://github.com/jupyter/jupyter-packaging.

For inspiration, we could also have a look at https://github.com/bloomberg/bqplot, https://github.com/jovyan/pythreejs, https://github.com/ellisonbg/ipyleaflet or https://github.com/matplotlib/ipympl.

With this and https://www.npmjs.com/package/c137.js shared by @TJKoury above, we should have everything we need already.

GorgiAstro commented 4 years ago

According to this post, it could be necessary to get a webpack of Cesium: https://github.com/jupyterlab/jupyterlab/issues/5566

A new webpack of Cesium will come soon: https://github.com/DigitalArsenal/webpack-cesium/issues/7

astrojuanlu commented 4 years ago

Oh wow, Jon Mease already worked on it :flushed: perhaps we'll have more luck this time!

GorgiAstro commented 4 years ago

Yes, but I could not find where he published his ipywidget. Maybe we can ask him in a chat or by tagging him here?

astrojuanlu commented 4 years ago

Hi @jonmmease! We noticed your comment from a couple of years ago about making a Cesium widget. @GorgiAstro found that there will be a new version of webpack-cesium soon. Do you have any code you can share so we can pick up from there and perhaps finish the work? Or do you think we should start from scratch?

jonmmease commented 4 years ago

Hi @astrojuanlu, thanks for reaching out. Sorry to say that I can't share the code, but that said it wasn't very far along. Most of the time spent was in tinkering to try to get cesium building as an ipywidget. Sounds like a webpack-cesium would make that much easier!

In terms of the widget itself, all that I got working was a proof-of-concept where I made a Python/JavaScript model that stored a single lat/lon/altitude coordinate. Updating that coordinate from python triggered the Cesium API to move the sphere around the earth.

Had I stuck with it, I was picturing adopting the approach that bqplot uses (https://github.com/bloomberg/bqplot), where you create a matching hierarchy of nested Python and JavaScript models. But that idea is as far as I got. Hope that helps a little. Good luck!

TJKoury commented 4 years ago

@jonmmease @astrojuanlu We have a new build that works great, and will be kept up to date with the latest in core Cesium.

As I am sure you are aware, the packing process is non-trivial, and in our case represents a significant investment to maintain. We are trying to keep it open for community use but have not decided on a licensing model yet.

Would any of the companies represented on this project consider sponsoring the work?

astrojuanlu commented 4 years ago

Thanks for chiming in @TJKoury! That's great news.

Would any of the companies represented on this project consider sponsoring the work?

poliastro and czml3 are a community effort, unfortunately the only support we can offer is integrating webpack-cesium in czml3 and try to make it work for a broader audience.

TJKoury commented 4 years ago

@astrojuanlu Understood. How do you think the community would respond to a 'non-commercial' license?

astrojuanlu commented 4 years ago

@TJKoury It's a good question. There are many forms of "non-commercial" licensing, for example what Qt does. If you decide use a copyleft license, we'll have to evaluate how to use it from czml3 (MIT, hence non-copyleft) without infringing the terms (and I might need a lawyer for this...). Other than that, I don't know how that can affect adoption.

TJKoury commented 4 years ago

Also, we are about to release a commercial package for the library behind Celestrak Orbit Viz.

TJKoury commented 4 years ago

After some consideration, I think we will probably just release it under MIT and ask for donations.

astrojuanlu commented 4 years ago

After some consideration, I think we will probably just release it under MIT and ask for donations.

That's great news :) Let me suggest https://opencollective.com/ for the donations part in case you don't know it. Looking forward for the release :muscle:

TJKoury commented 3 years ago

Here it is, C137.js.

Decided to go with CC BY 4.0 as freeware, since source-wise it is already open.

Left some crypto addresses at the bottom if people are feeling generous.

astrojuanlu commented 3 years ago

Awesome @TJKoury, thanks a lot! We'll soon try to figure out how to use, if @jonmmease doesn't beat us to it :)

TJKoury commented 3 years ago

It should be fairly straightforward, use an es6 import any bundler will take care of the rest. Let me know if you have any issues.

jonmmease commented 3 years ago

... if @jonmmease doesn't beat us to it :)

No risk of that for the foreseeable future :slightly_smiling_face:

astrojuanlu commented 3 years ago

Seems like JupyterLab 3.0 and webpack 5.0 will change some things? See lengthy discussion at https://github.com/jupyterlab/jupyterlab/issues/7468, keyword "Module Federation" https://module-federation.github.io/

TJKoury commented 3 years ago

@astrojuanlu It will not change the need to pack it up, or mess with our implementation. The issue is not dependency management, it is trying to load external resources like workers at runtime as if they were compile time assets.

astrojuanlu commented 3 years ago

I edited the first comment to add some extra information.

petrushy commented 3 years ago

Hi, was playing a bit with this, would like to get the old CesiumWidget up and running again.. Did a test with c137.js and cookiecutter widget, now showing the widget nothing else. https://github.com/petrushy/CesiumWidget3/tree/c137_version

I would however like to transition to the "official" cesium where all steps are reproducable, but c137.js was indeed very smooth to get running.

astrojuanlu commented 3 years ago

That's great news @petrushy ! I think it's pointless to have two widgets, so one option would be that we remove our widget.py from czml3 and we lend a hand with the maintenance of CesiumWidget3, what do you think?

TJKoury commented 3 years ago

Thanks for the vote of confidence @petrushy, and make sure to check out our soon-to-be decentralized notebook app, mapshot.app. Still in beta but you get the idea.

petrushy commented 3 years ago

@astrojuanlu I am very happy to collaborate on this, the one currently in czml is working but does not fully meet the needs of a widget for other purposes. I am thinking to also add a bit of higher level easy-plotting for quick visualizations.

@TJKoury , sorry I may express myself unclear, I do not see a future using closed source, your package is very nice and the example was a quick confirmation of that, but we need unfortunately to have transparency of the full process of the items that goes into it.

TJKoury commented 3 years ago

@petrushy No problem! The source is available, it’s just the build process that is proprietary. Seems like this project does not have issues using libraries built with closed-source IDE’s or operating systems, but if this is where you draw the line I understand.

TJKoury commented 2 years ago

It's open source now, Apache 2.0 licensed:

Github - c137.js

astrojuanlu commented 2 years ago

Thanks a lot @TJKoury !

By the way, this discussion is very relevant: https://github.com/jupyter/notebook/issues/6210 basically we are in the same situation as many other extensions. Doing what czml3 does in Classic Notebook is easy, but it turns out that doing it in JupyterLab requires an entirely different set of skills. I added my two cents to that discussion, and it looks like there are some ideas to make tinkering with JupyterLab easier. But if we want this migration to happen in the short term, we will have to bite the bullet and figure out how to do it. Undoubtedly, having c137.js open sourced makes everything easier.

TJKoury commented 2 years ago

@astrojuanlu Sounds good, let us know if there's anything on our end that we can do to help.

petrushy commented 2 years ago

It's open source now, Apache 2.0 licensed:

Many Thanks!!!

astrojuanlu commented 2 years ago

PSA: I'm learning webpack. I might tackle this myself 😉

astrojuanlu commented 2 years ago

Inspiration: https://github.com/damianavila/RISE/pull/605

SidharajYadav commented 2 years ago

how i contribute please guide me

dfreeman06 commented 2 years ago

Started work here starting from a JupyterLab extension cookiecutter: https://github.com/dfreeman06/czml3/tree/gh61_support_jupyterlab with strong assistance from @bollwyvl.

The webpack integrates with the cesium source directly but takes care to keep the various modules loading asynchronously so try and keep the bundles separated. Currently there is a rough wrapper around the Cesium Viewer, Clock, Camera, and CZMLDataSource. But it should not be too much effort to pull in the data sources.

This also is using the main Cesium.d.ts typings to generate the entity json schema (CZML json spec) which could be used to help address https://github.com/poliastro/czml3/issues/81.

I would love to get a PR back here so it would be great to get some more eyes on it and find the right path to integrate.

bollwyvl commented 2 years ago

@dfreeman06 you're a madman :exploding_head: glad you got it out there, happy to help move it forward.

unless the implementation shown is entirely different than the one we looked at the other day, here are some big ticket items to consider:

replaces the attrs parameter system with ipywidgets and traitlets
- it might be possible to emulate the existing API, but there would be new primitives, as described
- more importantly, it becomes possible to get live data back from cesium (e.g. camera position, selections)
- it might eventually be possible to integrate with other widget-based geometry systems such as pythreejs, jupyter-cadquery and ipyleaflet
- this could be particularly interesting for large datasets, as some of the above already work over the binary buffer system that, at its core, efficiently bridges numpy arrays to the browser
increases complexity of the build
- there is some dark magic in the .d.ts->schema.json as well as webpack stuff in general
- we've been using doit to handle these inter-language dependencies
ties more closely to the jupyterlab/lumino stack, but this includes
- retrolab (soon too be notebook)
- jupyterlite
unlike the RISE work, which has to deal with the very ill-tempered reveal.js, modern cesiumjs (~1.90) is very hygenic aside from one window-level global URL, which can be forgiven)
- what you're seeing in that screenshot are not iframes, but rather all in-dom, which is what enables properly sharing cameras, etc.
- there is some complexity when rendering the same viewer twice, but we'll be able to figure something out
- confidently upgrading to a new cesium would be a matter of re-solving the js packages and re-building

TJKoury commented 2 years ago

Some of those issues are alleviated if you swap out our single-file open source distro, c137.js, for Cesium. We made it in part to deal with the issues you mentioned. You can clone the repo and use the build process with the Cesium code, if you want to customize.

astrojuanlu commented 2 years ago

This is awesome, thanks a lot @dfreeman06 and @bollwyvl! Happy to review whenever you're ready to submit a PR.

bollwyvl commented 2 years ago

c137.js

We had a look at c137, and indeed may emulate some of it in separating the build into several packages, with one just being the cesium assets. Through no shortcoming of c137, a lab extension is going to go through the lab-specific build chain, and would still need webpack tricks.

While a bigger build-time download, by depending on cesium directly, we also gain access to the raw upstream cesium assets on disk, which is helpful for codegen, gathering licenses, and the ability to provide/manage a shared/federated cesium for other extensions. So also depending on c137 would actually end up being even bigger, and more fragile vs version drift.

whenever you're ready to submit a PR.

Well, I didn't know we were going to be sharing this in this state just yet... So there are still a lot of hanging bits, and will take some doing to get it into shape!

TJKoury commented 2 years ago

Makes sense, let us know if you have any questions about the parts of the build process you may decide to use.

petrushy commented 2 years ago

Looks great!

astrojuanlu commented 1 year ago

Well, JupyterLab 4.0 is out and Classic Notebook 7.0 will be out soon based on JLab, which means that, if I understood correctly, czml3 is about to stop working everywhere.

To this day I still don't understand the various ways to extend JupyterLab, and even though my JavaScript/TypeScript skills are getting better, the Jupyter docs seem to get more complicated at a faster pace, so there's no way I catch up. In addition, I stepped down as maintainer of poliastro in December 2022 https://www.poliastro.space/blog/2022/12/21/juan-luis-steps-down/, so I'm not actively maintaining czml3 anymore.

Still, efforts are welcome to fix this issue "the right way", whatever that is.

Interesting links:

astrojuanlu commented 11 months ago

Potentially helpful https://blog.jupyter.org/anywidget-jupyter-widgets-made-easy-164eb2eae102

astrojuanlu commented 5 days ago

Further inspiration https://github.com/jupyterlab/jupyterlab/issues/14590

poliastro / czml3

Support JupyterLab #61