search

ecmwf / earthkit

Python tools to work with weather and climate data
https://earthkit.readthedocs.io/
Apache License 2.0
40 stars 3 forks source link

JupyterLite demo #41

Open juntyr opened 5 months ago

juntyr commented 5 months ago

TLDR: Try earthkit in your browser without installing anything here

I'm a PhD student working on the online data compression laboratory for ESiWACE3. After I first heard about EarthKit yesterday, I thought I could try to add it to my online lab. In short, the online lab uses a custom Pyodide build (Python running inside your web browser using WebAssembly) together with JupyterLite (JupyterLab but using Pyodide) and adds support for several meteorology-specific Python packages that are not yet (publicly) supported for WebAssembly. The key selling point of the lab is that it is immediately accessible to anyone without them needing to install anything. As work on the online lab and its connected publication is ongoing, some of the magic behind the scenes is not open-source quite yet, but I wanted to share the small demo with you.

If you'd be interested in hosting a larger online demo for earthkit, e.g. to provide runnable documentation examples, I'd be happy to help you setup a similar Pyodide build.

tlmquintino commented 5 months ago

thanks for this. it is really interesting. I get an error while running the code in the link, but generally speaking I think we would be interested in this, especially to have some runnable earthkit demos. Please do get in touch with us.

sandorkertesz commented 5 months ago

Thank you for creating this demo. It worked for me perfectly well (used firefox on a mac)!

I just wonder how you install earthkit-data into the environment, because you seem to have the binary dependencies (e.g. eccodes for GRIB handling) on top of the Python packages. Do you use conda?

juntyr commented 5 months ago

You are asking about the magic behind the scenes :)

For Pyodide, all dependencies are compiled to WebAssembly. For pure Python packages, nothing has to be done. To get eccodes working, I compile libeccodes to WebAssembly as a shared library (note that WASM32 is a 32bit system, eccodes only supports 64bits, but disabling its check for that "just works") and slightly patch the gribapi/bindings.py to find the library (and eccodes definitions) in an unusual path.

At the moment, the Pyodide project manually maintains a huge variety of non-pure Python packages to make sure that they can be used inside the browser. In my online lab, I have simply extended this collection with several more packages. In the long run, PyPi will add support for developers to publish WASM-compiled versions there directly and WebAssembly will just become another platform that Python packages may or may not support.

If a demo such as this one works, WebAssembly support could be upstreamed back into e.g. eccodes and I could try to get it into the roster of Pyodide-supported packages (e.g. so that anyone using the "Try Jupyter" page, which also uses JupyterLite, could just import eccodes there).