xeus-sqlite kernel for Jupyterlite

[psychemedia]

I note from the https://github.com/jupyterlite/ org that several of the demo JupyterLite kernels are xeus-flavoured kernels.

What would be involved in porting this xeus-sqlite kernel to JupyterLite?

Is there a simple set of steps that could be described that would provide guidance for porting any xeus-kernel to jupyterlite (presumably assuming appropriate WASM compiled or runnable packages that are required are available)?

[marimeireles]

Hey @psychemedia thanks for opening this issue. I'm actually not super up to date to the state of this, but I think @DerThorsten can shed a light on us both. You know anything about this? :)

[DerThorsten]

Hi @psychemedia, The first step would be similar as for xeus-lua:

• get a static library of SQLite compiled to wasm.
• create a main which creates the emscripten bindings: This is basically this file here (just replace xlua::interpreter with the xeus-sqlite interpreter.)
• add the appropriate CMake code to build this file: Either you copy-paste the code from the xeus-lua kernel or create a fresh project with the xeus-cookiecutter

Once this builds one needs to create the typescript glue code. For xeus-lua this is done here This is still a bit messy:

• The easiest is to just copy the complete jupyterlite/xeus-lua-kernel repo
• change the Dockerfile st. it builds SQLite for wasm and the xeus-sqlite kernel for wasm
• search for all lua's in the repo and replace it with SQLite

At some point we should refactor the typescript glue code - part st. one can reuse more code.

[DerThorsten]

@psychemedia feel free to ping me if you need more info =)

[psychemedia]

Thanks. So could I also just take the sqlite.wasm from eg https://cdnjs.com/libraries/sql.js and use that rather than building my own?

[DerThorsten]

@psychemedia the sqlite.wasm file is the "exetuable" and not the library. For static libraries the extension should *.a. (I think in the past, static libraries could also en up in *.bc files)

https://emscripten.org/docs/compiling/Building-Projects.html

The file output from make might have a different suffix: .a for a static library archive, .so for a shared library, .o for object files (these file extensions are the same as gcc would use for the different types). Irrespective of the file extension, these files contain something that emcc can compile into the final JavaScript + WebAssembly (typically the contents will be wasm object files, but if you build with LTO then they will contain LLVM bitcode).

Having a quick look at the makefile, I supsect this might be the library itself.

[DerThorsten]

I would suggest using the makefile in the dockerfile as we do for xeus-lua:

We clone the repo here: https://github.com/jupyterlite/xeus-lua-kernel/blob/f53e035f17066388a6052e94ffde68398d2ab7ef/Dockerfile#L44 and build the static library with the makefile (this gives us a *.a file)

And link against this file here: https://github.com/jupyterlite/xeus-lua-kernel/blob/f53e035f17066388a6052e94ffde68398d2ab7ef/Dockerfile#L152

Also you need to include the header files as we do here: https://github.com/jupyterlite/xeus-lua-kernel/blob/f53e035f17066388a6052e94ffde68398d2ab7ef/Dockerfile#L151

[psychemedia]

Ah, ok... 30+ years since I played with compiling C!

For my ref, SQLite sql.js WASM build here: https://github.com/sql-js/sql.js/blob/master/Makefile

[bollwyvl]

Good stuff: it's early enough that everybody wins the more experiments are attempted.

Once the xeus approach gets more firmed up (e.g. via mamba/boa or what have you) and is less-copy-pasta and more configuration, this will probably be a very good approach. But I'll agree "just update this dockerfile and makefile," is probably going to pre-suppose a fair amount of experience with the xeus stack.

the sqlite.wasm file is the "exetuable"

indeed: perhaps there is a place for a xeus-wasm metakernel that would allow one to drop in an existing wasm, write a few functions (maybe in assemblyscript?), and now be able to speak a new language entirely (maybe at the same time as other languages). this is way outside the scope of the core lite project, but with efforts like wasmer's wapm, etc. being able to use "canonical" upstream wasm distributions rather than building everything from source might be very good.

For my ref, SQLite sql.js

As mentioned over there, I've done a lot of the initial scut work of wrapping sql.js on @gt-coar/jupyterlab-sqlite3, but it's more targeted at offering the sharedPackage and a promise-based API to other extensions rather than a kernel. However, if uncomfortable with docker and makefiles, but more ready to deal with webpack nonsense, one probably could repuprose that code to offer a kernel-like interface.

One part of sql.js that I would know how to maintain in the xeus stack (or even which language it would add) is the ability to embed js functions into the language. This could have very interesting effects, especially when used in triggers.

[psychemedia]

@bollwyvl Ooh... embedding js functions sounds interesting... I just learned about custom py sqlite functions for scalars, aggregations and (via vtfunc) table returning end-use defined functions, and had a lot of fun seeing what I could do quite quickly...

Re: the makes'n'webpack, that is out of my ken, though I'm reasonably comfortable trying to tweak Dockerfiles. A longer term goal of being able to easily drop a wasm package into a kernel template / proxy, with some very easily specified glue to make sure things hook up correctly sounds like some kind of nirvana...!

[DerThorsten]

I hacked a quick prototype: https://github.com/DerThorsten/jupyterlite_xeus_sqlite/ which seems to work in principal: But so far the only FileSystem which is working is a temporary one. This means all databases will vanish when the kernel is closed. It is compiled with vega support but I have not tested this yet. @jtpio jupyterlite_xeus_sqlite and jupyterlite_xeus_lua are basically the same repo, only the kernel logos, names and Dockerfiles differ

[jtpio]

perhaps there is a place for a xeus-wasm metakernel

Maybe a first step would be to refactor the common glue code used in the Lua, Wren and SQLite xeus-based kernels?

Which is something that should be happening in https://github.com/jupyterlite/xeus-kernel some time soon.

[psychemedia]

JupyterLite sqlite kernel now available, based on this kernel; see https://github.com/jupyterlite/xeus-sqlite-kernel