Install on Pyodide/JupyterLite

[deeplook]

It would be really cool to run leafmap on Pyodide and hence JupyterLite using micropip.

I gave it a try and found basically two issues with non-existing Python 3 source wheels (needed by micropip) for pycrs and pyshp on PyPI for which I've created these tickets (but there might be wheels for them hosted somewhere else): https://github.com/karimbahgat/PyCRS/issues/56 and https://github.com/GeospatialPython/pyshp/issues/217. And here-map-widget-for-jupyter can be installed, too, but still has some issue with the JupyterLab extension, as described on https://github.com/heremaps/here-map-widget-for-jupyter/issues/25. Folium can be installed in a somewhat complicated manner, see the notebook pyolite folium.ipynb on JupyterLite.

Not sure if anything else is needed... ;)

[giswqs]

@deeplook Thanks a lot for looking into this. It would be super awesome to have leafmap run on JupyterLite. Are the source wheels required by micropip the same as the one hosted on PyPI? I just built the pyshp wheel and made it available at https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl. However, it seems micropip failed to fetch the wheel. I also tried uploading the wheel to JupyterLite and installed it, but it failed as well. Any advice? @jtpio @martinRenou

await micropip.install('https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl')

await micropip.install('pyshp-2.1.3-py3-none-any.whl')

[deeplook]

I'm pretty new to this, too. The geojson package which is not preinstalled in JupyterLite can be installed with either of these lines (and the wheel was created in August 2019, see https://pypi.org/project/geojson/#files ;):

await micropip.install("https://files.pythonhosted.org/packages/e4/8d/9e28e9af95739e6d2d2f8d4bef0b3432da40b7c3588fbad4298c1be09e48/geojson-2.5.0-py2.py3-none-any.whl")

await micropip.install("geojson")

There is also this description for creating a Pyodide package, but I need to digest more of it...

And then, I read somewhere in the Pyodide issues on Github (I believe) that micropip will load the wheels using HTTP range requests, which maybe not all servers do support...

[jtpio]

Yes it would be interesting to double check installing pyshp from another server.

The content type seems to be differing:

$ curl -i https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl
HTTP/1.1 200 OK
Date: Wed, 23 Jun 2021 20:53:41 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Vary: User-Agent,Accept-Encoding
Last-Modified: Wed, 23 Jun 2021 17:05:46 GMT
ETag: "9190-5c571eaedc79d"
Accept-Ranges: bytes
Content-Length: 37264
Content-Type: application/x-troff-man

and

$ curl -i https://files.pythonhosted.org/packages/e4/8d/9e28e9af95739e6d2d2f8d4bef0b3432da40b7c3588fbad4298c1be09e48/geojson-2.5.0-py2.py3-none-any.whl
HTTP/2 200
last-modified: Wed, 26 Feb 2020 17:59:43 GMT
etag: "183b6f5ccb61818543f29c62a47eff16"
content-type: binary/octet-stream
server: UploadServer
cache-control: max-age=365000000, immutable, public
accept-ranges: bytes
date: Wed, 23 Jun 2021 20:55:48 GMT
age: 1251652
x-served-by: cache-sea4481-SEA, cache-hhn4082-HHN
x-cache: HIT, HIT
x-cache-hits: 1, 1
x-timer: S1624481749.825108,VS0,VE1
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
x-robots-header: noindex
access-control-allow-methods: GET
access-control-allow-origin: *
content-length: 14839

[deeplook]

There is also this https://github.com/pyodide/pyodide/issues/1653#issuecomment-869046830 in https://github.com/pyodide/pyodide/issues/1653 linking such installation issues with coors and specific browser versions.

[giswqs]

@deeplook This is great to know. Should we wait for JupyterLite to incorporate the latest pyodide?

[jtpio]

@giswqs @deeplook this PR updates to the latest (dev) version of Pyodide: https://github.com/jtpio/jupyterlite/pull/178

Would you be able to test on the RTD preview? https://jupyterlite--178.org.readthedocs.build/en/178/_static/lab/index.html

[deeplook]

@jtpio Sorry, but I seem to wait forever to import micropip there.

[jtpio]

Ah, probably something that changed in pyodide?

[RandomFractals]

well, I got it to work with a custom leaflet map renderer in vscode: https://github.com/RandomFractals/vscode-leaflet#pyolite-notebook-example

only using Pyolite to load geoJSON tho :(

[Niko-La]

anyone get leafmap to install via pywheel or micro on jupyterlite. this would be great to skip the binder builds. I believe lot of new features are coming in pyodide that should make it easier.

[deeplook]

@giswqs Meanwhile it is possible to run geopandas, shapely, fiona and gdal inside pyodide, which seems like providing a big step further in running more of leafmap inside the browser! See https://github.com/pyodide/pyodide/issues/1569 and https://github.com/pyodide/pyodide/pull/3213 (with a test using pyscript in the comments of the latter).

[giswqs]

@deeplook This is awesome! I will look into it this weekend. PR is also welcome!

[Niko-La]

@deeplook this is great news. @giswqs leafmap would be awesome on pyscript, just in time as the binder builds are now heavy and take a while.

pyscript map html file & demo: https://github.com/ryanking13/geopandas-pyodide-test/blob/main/index.html https://ryanking13.github.io/geopandas-pyodide-test/

@ryanking13 thanks for this poc.

[ryanking13]

@ryanking13 thanks for this poc.

Glad to hear that it was helpful :)

Please don't hesitate to ping me or open an issue in Pyodide if you have any help or find a bug in using geopandas in Pyodide.

[giswqs]

I tried to install leafmap with piplite. It seems there are several dependencies that do not have a wheel, such as gdown, pycrs. Let me see if I can build the missing wheels.

[giswqs]

I was able to build wheels for gdown and pycrs. However, the other three dependencies (tornado, pyzmq, and argon2-cffi-binders) do not have noarch wheels. piplite does not accept noarch wheels. Any advice?

await piplite.install('https://giswqs.github.io/py-wheels/gdown-4.5.3-py3-none-any.whl')
await piplite.install('https://giswqs.github.io/py-wheels/PyCRS-1.0.2-py3-none-any.whl')

[giswqs]

tornado, pyzmq, and argon2-cffi-binders appear to be dependencies of notebook. I might need to create a leafmap-lite package without the notebook dependency.

[giswqs]

I created a new package called leafmap-lite without the jupyterlab and ipysheet dependencies (they require tonado, which can't be installed in jupyterlite yet). Excited to see leafmap finally works with JupyterLite.

https://giswqs.github.io/leafmap-jupyterlite

import piplite
await piplite.install('leafmap-lite')
await piplite.install('leafmap', deps=False)

[jtpio]

Really nice!

without the jupyterlab and ipysheet dependencies (they require tonado, which can't be installed in jupyterlite yet)

Is there any reason to depend on jupyterlab? Usually widgets don't need that dependency. For ipysheet probably the same could be done (trimming dependencies).

[giswqs]

@jtpio Thank you for the suggestion. I have made jupyterlab and ipysheet as optional dependencies, now leafmap can be installed easily using piplite.

Try it out at https://demo.leafmap.org

%pip install -q leafmap

https://user-images.githubusercontent.com/5016453/203998312-e0a9d3b3-7f8f-43d9-b4f8-99cc1478601f.mp4