changes for jupyterlite-kernel

[DerThorsten]

• added new target for jupyterlite kernel
• added FindSqlite3.cmake in dedicated cmake_emscripten folder:
  • this is needed since we use a non-custom installation of sqlite for the wasm build
  • cmake_emscripten is only included in wasm build
• Added new magic commands for wasm build:
  • FETCH <URL> <FILENAME> : download a file / sqlite database and store it at the filename
  • SET_IDBFS_DIR <DIR>: set the directory which is "cached" / "persisted" to the IndexedDB
  • PUSH_TO_IDBFS: 'flush' content of persisted directory to IndexedDB
  • tiny bugfix in create since the index of the used token was off by one

[DerThorsten]

@marimeireles the sql installation comes from the SQLiteCpp - installation. This gives us the library and the headers, but is does not ship a proper cmake integration, this is why we need that here. We could very easy write this FindSqlite thingy from scratch instead of using the copy pasted one since it only needs to work for the emscripten case and nothing else

[DerThorsten]

but @marimeireles we can also wait a bit with mering this since we are currently changing xeus itself a bit

[marimeireles]

but @marimeireles we can also wait a bit with mering this since we are currently changing xeus itself a bit

Not a problem for me, we can merge it now and then do a new PR once the changes upstream are in.

This gives us the library and the headers, but is does not ship a proper cmake integration,

I see. Cool. Was just wondering! :) Thanks for your work here @DerThorsten.

[psychemedia]

This looks really useful :-) But if the database is large it means the browser will also take quite a hit memory wise, and network / download time wise.

An alternative approach for accessing large SQLite databases from in-browser SQLite engines is described in Hosting SQLite databases on Github Pages. To quote that post, it uses "a virtual file system that fetches chunks of the database with HTTP Range requests when SQLite tries to read from the filesystem: sql.js-httpvfs."

[marimeireles]

@psychemedia, thanks for the input :)

I'll merge this PR and fix the CI on a next one.

[psychemedia]

@DerThorsten Do you have an example of using the magics to download and query data from a sqlite db located at a particular URL?

If I just try to run the %FETCH magic using this repo's Binder link, I get an error:

Error: Load a database to run this command.

If I create a database, the %FETCH runs without delay (too short a time for it to be blocking-running as it waits for the file to download) and then when I try the %LOAD I get an Error: The path doesn't exist.

[DerThorsten]

@psychemedia this feature only works in case of a wasm build (ie JupyterLite ) Also it is not yet integrated in the jupyterlite kernel repo

But should not be too complicated to make the changes in the kernel repo by changing the Dockerfile https://github.com/jupyterlite/xeus-sqlite-kernel/blob/c5fc770610453f0af80b6043fd430d96f9453a0d/Dockerfile#L188

[psychemedia]

Ah, ok, thanks... I must have misread the other thread that referred to this one..