Closed Vipul-Cariappa closed 10 months ago
Did some investigation.
The above errors mentioned are for undefined symbols. If you look at the symbols they all contain nlohmann
. I know that nlohmann_json
is one of the dependencies of xeus
. So, I thought linking to nlohmann_json
was not taking place. But it looks like nlohmann_json
is a header-only library, implying linking is not necessary.
There are also two other warnings: "function_name" may need to be added to EXPORTED_FUNCTIONS if it arrives from a system libraryand
warning: function signature mismatch`. I guess I need not worry about them because they are warnings, not errors.
Happy New Year !
The error is due to an ABI mismatch in nlohmann json, meaning you're linking with dependencies that link with different versions of nlohmann_json. @DerThorsten I think we had a similar issue that you fixed recently, right?
Happy New Year !
The error is due to an ABI mismatch in nlohmann json, meaning you're linking with dependencies that link with different versions of nlohmann_json. @DerThorsten I think we had a similar issue that you fixed recently, right?
Yes, exactly the same issue. Fixed by (re)-building everything with the same version of nlohmann_json
Fixed by (re)-building everything with the same version of nlohmann_json
Thanks for the advice. I got it working. But there was one other issue I faced while building.
Could not find a package configuration file provided by "doctest"
Look at https://github.com/Vipul-Cariappa/KariLang-Kernel/actions/runs/7396742302/job/20122493278
Which I fixed by compiling the doctest
using the following
mkdir -p /opt/doctest/build
git clone https://github.com/doctest/doctest.git /opt/doctest/src
cd /opt/doctest/src
git checkout v2.4.11
cd /opt/doctest/build
emcmake cmake ../src/ -DCMAKE_INSTALL_PREFIX=/custom_sysroot
emmake make -j2 install
Then add the following command line argument for the cmake
build of xeus
and xeus-lite
-Ddoctest_DIR=/custom_sysroot/lib/cmake/doctest
I would like to know if it is possible to compile without doctest
, by using some flag like -D NO_DOCTEST=1
? Because doctest
is used for testing, and that can be skipped here ...?
Should I open a PR at xeus-cookiecutter repository, with the above updates for the GitHub emscripten_wasm_docker
workflow? I believe emscripten_wasm
workflow should also be changed to look exactly like the emscripten_wasm_docker
one. Because it requires that all the libraries be compiled with the correct versions. Do we plan to keep both emscripten_wasm_docker
and emscripten_wasm
which are identical, or remove one of them?
Question unrelated to this issue: How do I test the wasm build?
When I make install
and open jupyter notebook, while selecting the kernel it given the following error
[Errno 2] No such file or directory: '/home/vipul/micromamba/envs/karilang-wasm-jupyter-test/bin/xkarilang'
If I change the kernel.json
file to point to the wasm file, then I get
[Errno 13] Permission denied: '/home/vipul/micromamba/envs/karilang-wasm-jupyter-test/bin/xkarilang.wasm'
I also installed jupyterlite-core
and ran jupyter lite init
and jupyter lite build --output-dir dist
and tested it used python's HTTP server. It does not include the wasm kernel.
Hi,
I would suggest that you just use the xeus from emscripten-forge. On emscripten forge we have the whole xeus-stack compiled to wasm already.
To run a jupyterlite kernel from a local build of a xeus kernel you can follow these instructions here. https://github.com/jupyterlite/xeus?tab=readme-ov-file#from-local-environment--prefix
The repo https://github.com/jupyterlite/xeus is a relatively new repo which should simplify the deployment / usage of xeus wasm kernels.
To run a jupyterlite kernel from a local build of a xeus kernel you can follow these instructions here. https://github.com/jupyterlite/xeus?tab=readme-ov-file#from-local-environment--prefix
I tried following those instructions; cmake
errors
Could not find a package configuration file provided by "xeus-lite"
So I run
micromamba create -n xeus-python-dev \
--platform=emscripten-wasm32 \
-c https://repo.mamba.pm/emscripten-forge \
-c https://repo.mamba.pm/conda-forge \
--yes \
"python>=3.11" pybind11 nlohmann_json pybind11_json numpy pytest \
bzip2 sqlite zlib libffi xtl pyjs \
xeus xeus-sqlite xeus-lite
i.e. included xeus-lite
in the installation.
But building jupyterlite files with
jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
Does not add my kernel to the final build. I am still not able to access my kernel from the browser.
To run a jupyterlite kernel from a local build of a xeus kernel you can follow these instructions here. https://github.com/jupyterlite/xeus?tab=readme-ov-file#from-local-environment--prefix
I tried following those instructions;
cmake
errorsCould not find a package configuration file provided by "xeus-lite"
So this happens when you try to build your own kernel? Do you have xeus-lite installed?
So I run
micromamba create -n xeus-python-dev \ --platform=emscripten-wasm32 \ -c https://repo.mamba.pm/emscripten-forge \ -c https://repo.mamba.pm/conda-forge \ --yes \ "python>=3.11" pybind11 nlohmann_json pybind11_json numpy pytest \ bzip2 sqlite zlib libffi xtl pyjs \ xeus xeus-sqlite xeus-lite
i.e. included
xeus-lite
in the installation.
So once you have installed xeus-lite (and the other dependencies) you can compiled your kernel with cmake.
But building jupyterlite files with
jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
Does not add my kernel to the final build. I am still not able to access my kernel from the browser.
Do you install the kernelspec like we do for all our other kernels? https://github.com/jupyter-xeus/xeus-sqlite/blob/main/CMakeLists.txt#L331
And is the name of kernel target the same name as mentioned in the kernelspec? https://github.com/jupyter-xeus/xeus-sqlite/blob/4a5f6ae12525f770a2c6cddb2edf1c8cf5415945/CMakeLists.txt#L288 https://github.com/jupyter-xeus/xeus-sqlite/blob/main/share/jupyter/kernels/xsqlite/kernel.json.in#L4
Also, if you want to take any dependency from emscripten forge you need to take emscripten 3.1.45
We changed a few things in the cookie-cutter recently, so maybe just try with running the cookie-cutter again https://github.com/jupyter-xeus/xeus-cookiecutter
So this happens when you try to build your own kernel? Do you have xeus-lite installed?
Yes. The error is generated when I try to compile/build.
No, xeus-lite
is not installed at this point in time. Look at the diff of jupyterlite/xeus#22. The instructions does not include xeus-lite
in its dependencies.
So once you have installed xeus-lite (and the other dependencies) you can compiled your kernel with cmake.
Yes. I am able to compile and .wasm
and .js
files are also generated.
Do you install the kernelspec like we do for all our other kernels? https://github.com/jupyter-xeus/xeus-sqlite/blob/main/CMakeLists.txt#L331
Yes. That portion of the CMakeLists.txt
file look similar. I am pasting the relevant portion of it below.
if (KARILANG_BUILD_EXECUTABLE OR EMSCRIPTEN)
# Configuration and data directories for jupyter and xeus-karilang
set(XJUPYTER_DATA_DIR "share/jupyter" CACHE STRING "Jupyter data directory")
# Install xkarilang Jupyter kernelspec
set(KERNELSPEC_DIR ${CMAKE_CURRENT_SOURCE_DIR}/share/jupyter/kernels)
install(DIRECTORY ${KERNELSPEC_DIR}
DESTINATION ${XJUPYTER_DATA_DIR}
PATTERN "*.in" EXCLUDE)
# Extra path for installing Jupyter kernelspec
if (XEXTRA_JUPYTER_DATA_DIR)
install(DIRECTORY ${KERNELSPEC_DIR}
DESTINATION ${XEXTRA_JUPYTER_DATA_DIR}
PATTERN "*.in" EXCLUDE)
endif ()
endif ()
You can have a look at the complete CMakeLists.txt here.
And is the name of kernel target the same name as mentioned in the kernelspec? https://github.com/jupyter-xeus/xeus-sqlite/blob/4a5f6ae12525f770a2c6cddb2edf1c8cf5415945/CMakeLists.txt#L288 https://github.com/jupyter-xeus/xeus-sqlite/blob/main/share/jupyter/kernels/xsqlite/kernel.json.in#L4
Yes. I am pasting the relevant portion of the files below
if(EMSCRIPTEN)
include(WasmBuildOptions)
find_package(xeus-lite REQUIRED)
add_executable(xkarilang src/main_emscripten_kernel.cpp )
target_link_libraries(xkarilang PRIVATE xeus-lite KariLang_language)
KARILANG_set_kernel_options(xkarilang)
xeus_wasm_compile_options(xkarilang)
xeus_wasm_link_options(xkarilang "web,worker")
endif()
{
"display_name": "KariLang 0.0.1 (xkarilang)",
"argv": [
"@KARILANG_KERNELSPEC_PATH@xkarilang",
"-f",
"{connection_file}"
],
"language": "KariLang",
"metadata": {"debugger": false
}
}
I am going to paste some additional information below, if it helps in any means.
Output of make install
❯ emmake make install
make: make install
[ 61%] Built target KariLang_language
[ 84%] Built target xeus-karilang-static
[100%] Built target xkarilang
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang/logo-32x32.png
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang/kernel.json
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang/logo-64x64.png
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang/logo-svg.svg
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/lib/cmake/xeus-karilang/xeus-karilangConfig.cmake
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/lib/cmake/xeus-karilang/xeus-karilangConfigVersion.cmake
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/bin/xkarilang.js
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/bin/xkarilang.js
-- Up-to-date: /home/vipul/micromamba/envs/xeus-python-dev/bin/xkarilang.wasm
kernel.json
contents
❯ ccat /home/vipul/micromamba/envs/xeus-python-dev/share/jupyter/kernels/xkarilang/kernel.json
{
"display_name": "KariLang 0.0.1 (xkarilang)",
"argv": [
"/home/vipul/micromamba/envs/xeus-python-dev/bin/xkarilang",
"-f",
"{connection_file}"
],
"language": "KariLang",
"metadata": {"debugger": false
}
}
Contents of the xeus-python-dev/bin
folder
❯ ls /home/vipul/micromamba/envs/xeus-python-dev/bin
bzip2.js bzip2recover.wasm pip py.test python python3.11 sqlite3 xkarilang.js xmylang.js xsqlite.js
bzip2recover.js bzip2.wasm pip3 pytest python3 python.bak wheel xkarilang.wasm xmylang.wasm xsqlite.wasm
We changed a few things in the cookie-cutter recently, so maybe just try with running the cookie-cutter again https://github.com/jupyter-xeus/xeus-cookiecutter
I tried the latest version. The compilation and generation of the .wasm
and .js
files take place, but I am not able to use it from the browser. That is, I am not able to select the kernel from the browser.
From my general understanding of stuff; I believe I should copy the .wasm
, .js
and kernel.json
files to a specific location, and add some entry in someother file to register them as a kernel. When I run jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
, this is supposed to happen automatically. But it is not happening in my case. Can I do it manually. If it is as simple as I described?
Note: Does it make sense to move the above few conversations to a new issue. This issue is concerned with compilation errors. But the last few conversation is about running the compiled output from jupyterlite. Its a suggestion.
jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
I managed to get it working.
I created a new mamba environment and installed jupyterlite_xeus
, using python -m pip install jupyterlite_xeus
. Then I ran jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
.
Previously I had installed jupyterlite_xeus
and jupyterlite-core
in the system's python's default scope. My guess is that jupyterlite_xeus
is not happy with the system's version of python and requires mamba's version of python and the environment.
Including these details, relating to the virtual environments in the documentation (README.md) would be helpful. I can create a new PR in a few minutes for it and you can have a look at it.
karilang
is my kernel.
I'm so sorry, for taking your time. And thanks for all the help.
The next step would be to publish the "lite" (wasm) version of the kernel. Where would I publish it, to conda forge? or to emscripten forge? Does conda forge support wasm builds?
I managed to get it working.
nice
I created a new mamba environment and installed
jupyterlite_xeus
, usingpython -m pip install jupyterlite_xeus
. Then I ranjupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX
.Previously I had installed
jupyterlite_xeus
andjupyterlite-core
in the system's python's default scope. My guess is thatjupyterlite_xeus
is not happy with the system's version of python and requires mamba's version of python and the environment.Including these details, relating to the virtual environments in the documentation (README.md) would be helpful. I can create a new PR in a few minutes for it and you can have a look at it.
karilang
is my kernel.I'm so sorry, for taking your time. And thanks for all the help.
No worries, this is a very complicated and not very well documented stack
The next step would be to publish the "lite" (wasm) version of the kernel. Where would I publish it, to conda forge? or to emscripten forge? Does conda forge support wasm builds?
there is "emscripten-forge" which is a self-made conda-forge-like thing but for emscripten-wasm builds. https://github.com/emscripten-forge/recipes/pulls
I am trying to create a kernel for a toy programming language I created (Language at https://github.com/Vipul-Cariappa/KariLang). I followed the instructions from xeus's documentation "Implementing a kernel". I used the xeus-cookiecutter to initialize the boilerplate code. I have marked the
with_wasm
flag asyes
. I looked at the GitHub actions the cookiecutter generated to figure out how to compile it for jupyter-lite (i.e. wasm).I am running the following command:
All the above commands run fine except for the
make
. It fails with long output.I am able to compile the kernel and run it with
xeus-zmp
(dedicated processes), but compiling it withxeus-lite
(run in the browser) fails. You can look at my complete implementation of the kernel at https://github.com/Vipul-Cariappa/KariLang-Kernel. And the log of the GitHub action which builds thexeus-lite
version of the kernel.I am assuming I am doing something wrong, or there is a bug on xeus's side. Any information/suggestions would be helpful.
Happy New Year to you all.