feat: Isomorphic node/browser usage

[jeswr]

Thanks again for creating this package!

I would like isomorphic usage across browser/node. As as far as I see it, this entails bunding the locateFile function in the package and exporting a different version based on whether the environment is browser or node. I will create a PR for this shortly.

[rla]

Thank you for these changes, especially Typescript support! I merged the current PR.

I have identified some improvements that maybe you could add.

Try reuse types from @types/emscripten. I had them in the original wrapper module. The types came from @types/emscripten package (but ignore everything else from the file): https://github.com/rla/npm-swipl-wasm/blob/fde83ac1aa262e277a521b344317797eaebdb7b4/src/index.ts

I'm not sure how high quality they are. These are not official from the Emscripten team. See https://www.npmjs.com/package/@types/emscripten

Browser version file locations must be dynamic or configurable. URL path /dist/swipl was hardcoded for the example, other users or use cases will unlikely have files in this location on the web server. Sadly the configurable way kind-of makes the wrapper less useful. This was the main reason why I stripped out the old wrapper https://github.com/rla/npm-swipl-wasm/blob/77a79c0eb6bb06e722ac8b4a40b3e9635648e444/src/locateFile-browser.ts#L3

Another option is to make it dynamic by obtaining the current location of the file (possibly by using window.location) and then using this to construct the paths of the files.

It would be nice to have an example how to use this with latest CRA or Webpack 5. That would make sure the current packaging is possible to use with bundlers.

[jeswr]

Try reuse types from @types/emscripten. I had them in the original wrapper module. The types came from @types/emscripten package (but ignore everything else from the file): https://github.com/rla/npm-swipl-wasm/blob/fde83ac1aa262e277a521b344317797eaebdb7b4/src/index.ts

I'll try and look into this in the next couple of days.

Browser version file locations must be dynamic or configurable. URL path /dist/swipl was hardcoded for the example, other users or use cases will unlikely have files in this location on the web server. Sadly the configurable way kind-of makes the wrapper less useful. This was the main reason why I stripped out the old wrapper ... It would be nice to have an example how to use this with latest CRA or Webpack 5.

That makes sense. In fact I suspect at the moment a bundler would just not include the data and wasm files at all so there is a reasonable amount of work to be done to get this package working for browser. It is probably worth looking into how something like wasm-pack works to achieve a clean browser bundle for its package (https://www.reddit.com/r/rust/comments/h85a9n/publishing_wasm_packages_to_npm_for_use_by_nodejs/).

In the meantime I'm still happy to add the window.location patch.

[josd]

Also to get it more isomorphic with native swipl could we have Foreign extensions (packages that rely on C/C++ code) plus gmp support (see Building the WASM version and Build the WASM version using Emscripten). It works at least in Eyebrow which was built with an very primitive script.

[jeswr]

Also to get it more isomorphic with native swipl could we have Foreign extensions (packages that rely on C/C++ code) plus gmp support (see Building the WASM version and Build the WASM version using Emscripten). It works at least in Eyebrow which was built with an very primitive script.

@josd This is indeed important to get working on our use case of running https://josd.github.io/eye/eye.pl; which, as you mentioned to me internally you have temporarily remove the Forgeign extension packages for as a workaround.

I think this is largely orthogonal to this issue so it might be worth opening up a new issue to make it easier to track.

[rla]

I added GMP to the build script but static extensions do not seem to work. I'm a bit confused whether I have to pass -DSTATIC_EXTENSIONS=ONmyself or it comes automatically somehow. Anyway, after building I don't seem to have working library(crypt). It's also not reported when running the Node example with check_installation:

swipl.prolog.query('check_installation.').once();

Output:
................................................ not present
Warning: See http://www.swi-prolog.org/build/issues/tcmalloc.html
Warning: library(archive) ...................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/archive.html
Warning: library(cgi) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/cgi.html
Warning: library(crypt) ........................ NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/crypt.html
Warning: library(bdb) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/bdb.html
Warning: library(double_metaphone) ............. NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/double_metaphone.html
Warning: library(filesex) ...................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/filesex.html
Warning: library(http/http_stream) ............. NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/http/http_stream.html
Warning: library(http/json) .................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/http/json.html
Warning: library(http/jquery) .................. NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/http/jquery.html
Warning: library(isub) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/isub.html
Warning: library(jpl) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/jpl.html
Warning: library(memfile) ...................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/memfile.html
Warning: library(odbc) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/odbc.html
Warning: library(pce) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/xpce.html
Warning: library(pcre) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/pcre.html
Warning: library(pdt_console) .................. NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/pdt_console.html
Warning: library(porter_stem) .................. NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/porter_stem.html
Warning: library(process) ...................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/process.html
Warning: library(protobufs) .................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/protobufs.html
Warning: library(semweb/rdf_db) ................ NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/semweb/rdf_db.html
Warning: library(semweb/rdf_ntriples) .......... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/semweb/rdf_ntriples.html
Warning: library(semweb/turtle) ................ NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/semweb/turtle.html
Warning: library(sgml) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/sgml.html
Warning: library(sha) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/sha.html
Warning: library(snowball) ..................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/snowball.html
Warning: library(socket) ....................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/socket.html
Warning: library(ssl) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/ssl.html
Warning: library(crypto) ....................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/crypto.html
Warning: library(table) ........................ NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/table.html
Warning: library(time) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/time.html
Warning: library(unicode) ...................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/unicode.html
Warning: library(uri) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/uri.html
Warning: library(uuid) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/uuid.html
Warning: library(zlib) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/zlib.html
Warning: library(yaml) ......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/yaml.html
Warning: Found 36 issues

Here is the build script: https://github.com/rla/npm-swipl-wasm/blob/2cfdc2e83dc603e5702d7990e91ad5d98d2ea973/docker/build-swipl.sh

Maybe @JanWielemaker knows what could be the issue?

[JanWielemaker]

Surely -DSWIPL_PACKAGES=OFF is enough to explain this. The instructions at https://swi-prolog.discourse.group/t/swi-prolog-in-the-browser-using-wasm/5650#build-the-wasm-version-using-emscripten-15 are kept up to date (they tend to change regularly, making more defaults depend on Emscripten).

[rla]

Surely -DSWIPL_PACKAGES=OFF is enough to explain this. The instructions at https://swi-prolog.discourse.group/t/swi-prolog-in-the-browser-using-wasm/5650#build-the-wasm-version-using-emscripten-15 are kept up to date (they tend to change regularly, making more defaults depend on Emscripten).

Thanks! Now it works. I have built published new version of the package.

[rla]

Browser version file locations must be dynamic or configurable. URL path /dist/swipl was hardcoded for the example, other users or use cases will unlikely have files in this location on the web server. Sadly the configurable way kind-of makes the wrapper less useful. This was the main reason why I stripped out the old wrapper ... It would be nice to have an example how to use this with latest CRA or Webpack 5.

That makes sense. In fact I suspect at the moment a bundler would just not include the data and wasm files at all so there is a reasonable amount of work to be done to get this package working for browser. It is probably worth looking into how something like wasm-pack works to achieve a clean browser bundle for its package (https://www.reddit.com/r/rust/comments/h85a9n/publishing_wasm_packages_to_npm_for_use_by_nodejs/).

In the meantime I'm still happy to add the window.location patch.

From what I see the wasm-pack seems to only work for Rust-based packages.

With Webpack we might just want to produce a clever locateFile function. Seems like the asset-loader should give us the file locations (and automatically copy them to the public folder) if you "import" .data and .wasm, similar to image files. Still needs a working code example. https://webpack.js.org/guides/asset-modules/

[josd]

I think this is largely orthogonal to this issue so it might be worth opening up a new issue to make it easier to track.

@jeswr it is resolved before I could open a new issue 🙂 Thank you very much @rla and @JanWielemaker!!

[jeswr]

I've got a WIP PR to resolve https://github.com/rla/npm-swipl-wasm/issues/1#issuecomment-1232907915 in https://github.com/rla/npm-swipl-wasm/pull/3; however, I don't think I will actually be able to make it that clean until swipl-web.js actually uses import/require to get these files (rather than fetch which it appears to be using at the moment) so we can then use them properly as asset modules like in the example in https://webpack.js.org/guides/asset-modules/ and the similar examples for rollup asset loading.

[JanWielemaker]

JavaScript is not really my expertise :cry: If you have suggestions for that stuff, please share (preferably as PRs). Otherwise I rely on @rla and others ...

I'm working on getting the Prolog test suite to work with the WASM version. That is closer to my expertise :smile:

[jeswr]

I'll try and take a look at some point soon (this week or next week) @JanWielemaker - can you point me to the code that is actually responsible for generating swipl.js and swipl-web.js to save me some digging?

[JanWielemaker]

The WASM specific code is in src/wasm. I think you might want to look at prolog.js there. The stuff that sets the compile and link flags is in cmake/EmscriptenTargets.cmake and cmake/ports/Emscripten.cmake

[rla]

@jeswr, @JanWielemaker I propose to split off any further development into a separate package (swipl-wasm-isomorphic?) and slim down this one as much as possible to make it just a channel to distribute swipl.js, swipl-web.js data and wasm files. Typescript, as is, with incomplete 3rd party emscripten types, isomorphic usage etc. makes things way too complex to some users. This repo should also be transformed to under SWI-Prolog org in GH.

@JanWielemaker, could you take look at docker/Dockerfile and *.sh scripts there whether it would be possible to integrate with SWI-Prolog CI and/or with build scripts.

[JanWielemaker]

See also https://swi-prolog.discourse.group/t/wiki-discussion-swi-prolog-in-the-browser-using-wasm/5651/139?u=jan

I'll have a look a the Dockerfile tomorrow (as well as the rest of this repo). Eventually I'd like to see this all nicely integrated. Doesn't seem trivial though and it is a little out of my comfort zone :smile:

[rla]

The main goal of this project Dockerfile is to normalize the build environment:

• Specify Emscripten version
• Specify ZLIB and GMP versions
• Specify SWI-Prolog commit
• Specify platform tooling (cmake, etc) coming from Debian stable

This makes build a lot more predictable and repeatable and allows easy building on Windows/MacOS (minus the difficulty of setting up Docker). I experienced lots of software instability during porting. Even the Dockerfile approach is not fully stable because ZLIB tends to remove their old releases. Compared to Prolog and JavaScript languages like C and C++ have order of magnitude more complexity with their build environments.

One good option might be turning the Dockerfile+scripts into one big script which can be run both in docker and the host machine. The secondary goal of the Dockerfile was to cache away download/git clone of dependencies over repeated build runs.

[JanWielemaker]

Agree. We do the same for the build for Windows using MinGW based cross-compilation. That is a lot more complicated ...

[jeswr]

I dug into this a little - there is a -sSINGLE_FILE option for emscripten that bundles the webassembly data into the .js file. I was able to do a POC on this here by adding the following EmscriptenTargets.cmake to the docker directory of this project

# Make console binaries runnable through Node.js.

set(WASM_NODE_LINK_FLAGS
    -s NODERAWFS=1
    -s EXIT_RUNTIME=1)
join_list(WASM_NODE_LINK_FLAGS_STRING " " ${WASM_NODE_LINK_FLAGS})

set_target_properties(swipl PROPERTIES
              LINK_FLAGS "${WASM_NODE_LINK_FLAGS_STRING}")

# Create the preload data containing the libraries. Note that
# alternatively we can put the library in the resource file and
# link the resource file into the main executable.

set(WASM_BOOT_FILE "${WASM_PRELOAD_DIR}/boot.prc")
add_custom_command(
    OUTPUT ${WASM_BOOT_FILE}
    COMMAND ${CMAKE_COMMAND} -E make_directory ${WASM_PRELOAD_DIR}
    COMMAND ${CMAKE_COMMAND} -E copy ${SWIPL_BOOT_FILE} ${WASM_BOOT_FILE}
    COMMAND ${CMAKE_COMMAND} -E copy_directory
                 ${SWIPL_BUILD_LIBRARY} ${WASM_PRELOAD_DIR}/library
    DEPENDS ${SWIPL_BOOT_FILE} prolog_home library_index
    VERBATIM)

add_custom_target(wasm_preload_dir DEPENDS ${WASM_BOOT_FILE})
add_dependencies(wasm_preload wasm_preload_dir)

# Build the browser-deployed binary with a bit different linker flags.

set(POSTJS ${CMAKE_CURRENT_SOURCE_DIR}/wasm/prolog.js)
set(PREJS ${CMAKE_CURRENT_SOURCE_DIR}/wasm/pre.js)

set(WASM_WEB_LINK_FLAGS
    -s WASM=1
    -s MODULARIZE=1
    -s EXPORT_NAME=SWIPL
    -s NO_EXIT_RUNTIME=0
    -s WASM_BIGINT=1
    -s ALLOW_MEMORY_GROWTH=1
    -s EXPORTED_FUNCTIONS=@${CMAKE_SOURCE_DIR}/src/wasm/exports.json
    -s EXPORTED_RUNTIME_METHODS=@${CMAKE_SOURCE_DIR}/src/wasm/runtime_exports.json
    -s SINGLE_FILE
    --preload-file ${WASM_PRELOAD_DIR}@swipl
    --pre-js ${PREJS}
    --post-js ${POSTJS})
if(MULTI_THREADED)
  list(APPEND WASM_WEB_LINK_FLAGS
       -pthread
       -s PTHREAD_POOL_SIZE=4)
endif()
join_list(WASM_WEB_LINK_FLAGS_STRING " " ${WASM_WEB_LINK_FLAGS})

add_executable(swipl-web ${SWIPL_SRC})
set_target_properties(swipl-web PROPERTIES
              LINK_FLAGS "${WASM_WEB_LINK_FLAGS_STRING}")
target_link_libraries(swipl-web libswipl)
add_dependencies(swipl-web wasm_preload)
set_property(TARGET swipl-web PROPERTY LINK_DEPENDS
         ${POSTJS} ${PREJS})

and then copying it into the docker container by adding before the last build-swipl step

COPY EmscriptenTargets.cmake /swipl-devel/cmake/EmscriptenTargets.cmake

---

This gets you closer to being able to bundle everything into a single Javascript file; however, the swipl-web.data file is still being produced and required in the generated files. Is it coming from swipl or an artifact of emscripten (I will try and find a way of bundling it as well if the answer is emscripten)

[jeswr]

Btw I went and found this off the back of reading this blog post.

[jeswr]

I tried commenting out the --preload-file ${WASM_PRELOAD_DIR}@swipl argument to avoid the generation of the .data file but then trying to run the generated Javascript file results in the error

[FATAL ERROR: at Fri Dec 23 00:06:15 2022
        Could not find system resources]
Aborted()

[jeswr]

Ah figured it needed to switch out --preload-file with --embed-file

[JanWielemaker]

Btw I went and found this off the back of reading this blog post.

Promising. The referenced blog is nice reading. The conclusions mention that the price is rather high for exceptionally large WASM files. We probably classify 😢 . One of the long term plans I am interested in is to get rid of most of the data file and manage that by loading required libraries lazily from a server. That might complicate a single distro for Node and browser even further? As is though, we pay over 1Mb for including library(chr), required by only a small minority of the applications. That is just one example.

[rla]

Is this true that we need a single-file bundle only to support Cloudflare workers? Or is there any other use case?

A working example of using Webpack bundler is in https://github.com/rla/npm-swipl-wasm/tree/master/examples/webpack. It does exactly what the article says by using 'asset/resource'. The wasm and data files are copied to the output directory and import statement of either of those returns a server path to the file. The files are lazily loaded. The webpack configuration is nearly trivial: https://github.com/rla/npm-swipl-wasm/blob/master/examples/webpack/webpack.config.js

I can think of a second type of bundling. Do you need to create eye reasoner bundle that includes SWI WebAssembly build but as a single file that includes everything? If it did not include everything, users would need to configure its underlying SWI instance which I guess makes it a bit difficult. Then it would be nice if this could be actually done when building the eye bundle. But this is not possible because of how Emscripten is supposed to load .wasm and .data?

As is though, we pay over 1Mb for including library(chr), required by only a small minority of the applications. That is just one example.

Possibly. Maybe it should go though the same locateFile interface of Emscripten as is used for .data and .wasm today? Then this would be the interception point for bundlers/loaders in different environments. A list of lazily loaded library files could be easily bundled in, loaded from URLs, or just loaded from filesystem, depending on the environment.

[jeswr]

Is this true that we need a single-file bundle only to support Cloudflare workers? Or is there any other use case?

The main reason I wanted it is so that we have a solution that "just works" in both node and browser; without needing to do any custom work with Webpack/rollup/etc. on the consumer end - and also not needing to have to do anything with locateFile for either browser or node. This is really useful for saving headaches in prototyping / POCs that we are doing.

Note that users of eye-js still have the option to use swipl-web or swipl-node from this package if they choose

But this is not possible because of how Emscripten is supposed to load .wasm and .data?

It is possible; and that is exactly what https://github.com/rla/npm-swipl-wasm/pull/8 does; it inlines the WebAssembly in the JavaScript file as base64 using the -sSingle command and embeds the .data file as a string using the --embed-file command.

So everything that you need is in the swipl-bundle.js file. No need for webpack, no need for locateFile.

[jeswr]

The conclusions mention that the price is rather high for exceptionally large WASM files.

@JanWielemaker - I've just discovered that this actually not a problem in most of the use cases for eye as if gzip content encoding applied when serving the js files it seems to make up for the bloat in file size (see https://github.com/eyereasoner/eye-js#browser-builds - and this example shows you how fast it "feels").

That said - directly gzipping the .wasm file brings it down to 739KB

[JanWielemaker]

Hi @jeswr, sounds good. I have a bit of a backlog in issues and tasks. I assume you can live with your branch for now?

[jeswr]

Yes - currently we just have a file containing the custom bundle in eye rather than importing swipl-wasm.

[rla]

Bundled variant is now there in the 3.1 release. While the file is pretty large it does not seem to cause excessive memory usage during load or big load-time slowdown.