Investigate new micropip features

[bollwyvl]

Problem

We still carry a lot of patches, managed as full-fat wheel builds, #27 adds more.

Proposed Solution

Investigate whether new micropip features make this more reasonable.

• [ ] micropip.add_mock_package
  • this may allow moving heavyweight wheels to simple JSON descriptions
  • unknown whether this properly ignores packages known to need to ship with multiple versions mocks, e.g. widgetsnbextension {x|y}
  • special cases could potentially be detected and overridden
• [ ] micropip.install(..., index_urls)
  • presumably these are still clobbered by pyodide-lock.json
  • would need to decide how any configured settings would interact with direct calling with e.g. %pip
• [x] loadPyodide options
  • [x] lockFileURL
  • fully replaces the default of the default ${indexUrl}/pyodide-lock.json
  • using existing known URLs, this could be used to generate a new composite lock file a given upstream
  • theoretically could be done at runtime with a data: URI (unknown)
  • getting valid package descriptions still blocked by https://github.com/pyodide/pyodide-lock/issues/9
  • [x] packages
  • presumably these are the distribution names as found in the lock file
  • generally won't work for complex dependencies that do need patching as above
    • could add an advisory --pyodide-preload-packages-blocklist for kernel deps
  • for the special case of notebooks, the list of required modules could be read from metadata

Additional context

extracted from #39

[bollwyvl]

The micropip options were a bit too complex to solve in the otherwise-small #103. While these could still be added to piplite configuration somehow, it may be moot if we can get a closed form, build-time solution that generated a well-formed pyodide-lock.json, currently under investigation.

[bollwyvl]

Ok, I've got an (unreleased) jupyterlite-pyodide-lock up to a working state. There are some temporary docs with demo up, which also includes a deployed lockfile and wheel.

Right now it works with well-known locations of firefox and chrom(e|ium) over either:

• a naive subprocess (sadly couldn't reuse much of the cpython gubbins, as they punt to open, etc)
• the WebDriver standard, via selenium
  • this is separate installable package in contrib

Due to the hard dependency on (alpha) pyodide-lock, and the general flakiness of browser automation, this should probably not land in pyodide-kernel itself any time soon, though likely should be mentioned in the appropriate docs.

Other techniques (nodejs, playwright, pyppetteer, an embedded qt browser) are certainly possible, and could be added via the jupyterlite_pyodide_lock.locker.v0 entrypoint, but... I personally have no interest in maintaining them.

[bollwyvl]

and of course not precious about where this actually gets released from, if it makes more sense for it to be in the org.