[help] Docs do not reflect current lib

[DavidGOrtega]

Hi!

I wanted to give a shot the library converting hnswlib-node without luck

System

My system is MacOS Ventura running in an M1.

node -v
v16.18.1

npm -v
8.19.2

emcc -v
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.34-git
clang version 17.0.0 (https://github.com/llvm/llvm-project.git a031f72187ce495b9faa4ccf99b1e901a3872f4b)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/emscripten/3.1.34/libexec/llvm/bin

cmake --version
cmake version 3.25.1

Conversion step

yarn add @emnapi/runtime

I have slightly modified the C++ example

em++ -O3 \
     -DNAPI_DISABLE_CPP_EXCEPTIONS \
     -DNODE_ADDON_API_ENABLE_MAYBE \
     -I./node_modules/@emnapi/runtime/include \
     -L./node_modules/@emnapi/runtime/lib/wasm32-emscripten \
     --js-library=./node_modules/@emnapi/runtime/dist/emnapi.js \
     -sEXPORTED_FUNCTIONS="['_malloc','_free']" \
     -o hnswlib.js \
     ./src/addon.cc \
     -lemnapi

./node_modules/node-addon-api/napi.h:4:10: fatal error: 'node_api.h' file not found
#include <node_api.h>
         ^~~~~~~~~~~~
1 error generated.
em++: error: '/opt/homebrew/Cellar/emscripten/3.1.34/libexec/llvm/bin/clang++ -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN --sysroot=/opt/homebrew/Cellar/emscripten/3.1.34/libexec/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -O3 -DNAPI_DISABLE_CPP_EXCEPTIONS -DNODE_ADDON_API_ENABLE_MAYBE -I./node_modules/node-addon-api ./src/addon.cc -c -o /var/folders/7q/gtp1y2js5xng03nkg0mvd6xm0000gn/T/emscripten_temp__q2otjvq/addon_0.o' failed (returned 1)

however Im missing the include and wasm-emscripten folders. Please note also that Im uisng emnapi.js instead of library_napi.js that seems not to be either.

Am I going in the right direction?

Thanks 🙏

[toyobayashi]

You need to install emnapi and @emnapi/runtime, and the headers and libs are in emnapi npm package

-I./node_modules/emnapi/include \
-L./node_modules/emnapi/lib/wasm32-emscripten

And if your code uses async work or TSFN, you need to use -pthread and -lemnapi-mt instead. See the "Multithread" part in README.

[toyobayashi]

I successfully built your repo and run the code in README.

npm install -D emnapi
npm install @emnapi/runtime

em++ -O3 \
     -DNAPI_DISABLE_CPP_EXCEPTIONS \
     -I./node_modules/emnapi/include \
     -L./node_modules/emnapi/lib/wasm32-emscripten \
     --js-library=./node_modules/emnapi/dist/library_napi.js \
     -sEXPORTED_FUNCTIONS="['_malloc','_free']" \
     -sALLOW_MEMORY_GROWTH=1 \
     -sPTHREAD_POOL_SIZE=4 \
     -sMODULARIZE=1 \
     -sEXPORT_NAME=hnswlib \
     -sSTACK_SIZE=2MB \
     -sDEFAULT_PTHREAD_STACK_SIZE=2MB \
     -pthread \
     -o hnswlib.js \
     ./src/addon.cc \
     -lemnapi-mt

// index.js

const hnswlib = require('./hnswlib.js')
const { getDefaultContext } = require('@emnapi/runtime')

hnswlib().then(Module => {
  const binding = Module.emnapiInit({ context: getDefaultContext() })
  console.log(binding)

  const { HierarchicalNSW } = binding

  const numDimensions = 8
  const maxElements = 10

  const index = new HierarchicalNSW('l2', numDimensions);
  index.initIndex(maxElements);

  for (let i = 0; i < maxElements; i++) {
    const point = new Array(numDimensions);
    for (let j = 0; j < numDimensions; j++) point[j] = Math.random();
    index.addPoint(point, i);
  }

  // saving index.
  index.writeIndexSync('foo.dat');

  const index2 = new HierarchicalNSW('l2', 3);
  index2.readIndexSync('foo.dat');
  const query = new Array(3);
  for (let j = 0; j < 3; j++) query[j] = Math.random();
  const numNeighbors = 3;
  const result = index2.searchKnn(query, numNeighbors);

  console.log(result);
})

$ node ./index.js
{
  L2Space: [Function: L2Space],
  InnerProductSpace: [Function: InnerProductSpace],
  BruteforceSearch: [Function: BruteforceSearch],
  HierarchicalNSW: [Function: HierarchicalNSW]
}
{
  distances: [ 0.011937220580875874, 0.08729938417673111, 0.1758333146572113 ],
  neighbors: [ 1, 9, 6 ]
}

[DavidGOrtega]

Thanks a lot for the support!!!

I have tried myself and I have an error

em++: warning: -pthread + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 [-Wpthreads-mem-growth]
cache:INFO: generating system asset: symbol_lists/9dddaab49b8f96d245cb06d940493df9b9fb2c10.txt... (this will be cached in "/opt/homebrew/Cellar/emscripten/3.1.34/libexec/cache/symbol_lists/9dddaab49b8f96d245cb06d940493df9b9fb2c10.txt" for subsequent builds)
cache:INFO:  - ok
wasm-ld: error: ./node_modules/emnapi/lib/wasm32-emscripten/libemnapi-mt.a(async.c.o): undefined symbol: emscripten_main_browser_thread_id
em++: error: '/opt/homebrew/Cellar/emscripten/3.1.34/libexec/llvm/bin/wasm-ld -o hnswlib.wasm -L./node_modules/emnapi/lib/wasm32-emscripten /var/folders/7q/gtp1y2js5xng03nkg0mvd6xm0000gn/T/emscripten_temp_fz_nfjgl/addon_0.o ./node_modules/emnapi/lib/wasm32-emscripten/libemnapi-mt.a -L/opt/homebrew/Cellar/emscripten/3.1.34/libexec/cache/sysroot/lib/wasm32-emscripten /opt/homebrew/Cellar/emscripten/3.1.34/libexec/cache/sysroot/lib/wasm32-emscripten/crtbegin.o -lGL-mt -lal -lhtml5 -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --allow-undefined-file=/var/folders/7q/gtp1y2js5xng03nkg0mvd6xm0000gn/T/tmpmw9jt_m_.undefined --import-memory --shared-memory --strip-debug --export-if-defined=malloc --export-if-defined=free --export-if-defined=_emscripten_thread_init --export-if-defined=_emscripten_thread_exit --export-if-defined=_emscripten_thread_crashed --export-if-defined=_emscripten_tls_init --export-if-defined=pthread_self --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=emscripten_dispatch_to_thread_ --export=_emscripten_thread_free_data --export=emscripten_main_runtime_thread_id --export=emscripten_main_thread_process_queued_calls --export=_emscripten_run_in_main_runtime_thread_js --export=emscripten_stack_set_limits --export=__get_temp_ret --export=__set_temp_ret --export=__wasm_call_ctors --export=__cxa_is_pointer_type --export=_emscripten_timeout --export-table -z stack-size=2097152 --initial-memory=16777216 --no-entry --max-memory=2147483648 --global-base=1024' failed (returned 1)

My machine is MacOS M1

[toyobayashi]

Oh, emscripten_main_browser_thread_id has been renamed to emscripten_main_runtime_thread_id in emscripten-core/emscripten#18872 after 3.1.32, but emnapi 0.35.0's prebuilt library is built with emscripten 3.1.32. You can try to use emscripten 3.1.32

[toyobayashi]

It only broke the prebuilt library which use the old symbol. As emscripten left the old name emscripten_main_browser_thread_id as macro, I think using CMake add_subdirectory("./node_modules/emnapi") and target_link_libraries(${YOUR_TARGET} "emnapi-mt") should also work.

[DavidGOrtega]

@toyobayashi working!

Seems to be using SharedArrayBuffer. I need enable cross-origin-isolation. Is there any way to avoid it?

Also, seems to be using worker. Can I avoid the use of a web worker?

[toyobayashi]

Try this: -DNAPI_HAS_THREADS and -lemnapi-basic

em++ -O3 \
     -DNAPI_DISABLE_CPP_EXCEPTIONS \
     -DNAPI_HAS_THREADS \
     -I./node_modules/emnapi/include \
     -L./node_modules/emnapi/lib/wasm32-emscripten \
     --js-library=./node_modules/emnapi/dist/library_napi.js \
     -sEXPORTED_FUNCTIONS="['_malloc','_free']" \
     -sALLOW_MEMORY_GROWTH=1 \
     -sMODULARIZE=1 \
     -sEXPORT_NAME=hnswlib \
     -sSTACK_SIZE=2MB \
     -o hnswlib.js \
     ./src/addon.cc \
     -lemnapi-basic

async work implementation in libemnapi-mt.a is relying on emscripten's pthread, it schedules async task in worker threads. If you link libemnapi-basic.a, async work is just single thread mock, it's implemented in JavaScript, no worker required.

[DavidGOrtega]

-sSTACK_SIZE=2MB \

is there any particular reason do you choose this stack size?

[toyobayashi]

Emscripten changed the default stack size in recent version, that cause some emnapi test fail so I add it. You can try to omit it see if there is any error in your program.

[DavidGOrtega]

I have seen that recommended for node is 8192. Thats why I asked you why 2MB. If it was a good number.

Why do I get a stack size error when optimizing: RangeError: Maximum call stack size exceeded or similar?
You may need to increase the stack size for [node.js](https://emscripten.org/docs/site/glossary.html#term-node-js).

On Linux and Mac macOS, you can just do NODE_JS = ['node', '--stack_size=8192'] in the [Emscripten Compiler Configuration File (.emscripten)](https://emscripten.org/docs/tools_reference/emsdk.html#compiler-configuration-file). On Windows, you will also need --max-stack-size=8192, and also run editbin /stack:33554432 node.exe.

[DavidGOrtega]

Seems to be using SharedArrayBuffer. I need enable cross-origin-isolation. Is there any way to avoid it?

Can I avoid SharedArrayBuffer?

[toyobayashi]

Thats why I asked you why 2MB. If it was a good number.

No some good reason, you can set it to any size you need.

Can I avoid SharedArrayBuffer

With -lemnapi-basic and no pthread, emscripten won't use SharedArrayBuffer as webassembly's memory.

[DavidGOrtega]

closing 🙏