Error: recursive use of an object detected which would lead to unsafe aliasing in rust

[maccman]

Unsure how to give more info, but getting this error sometimes.

[maccman]

Also this:

[DawChihLiou]

Hey @maccman, could you provide me with a file of the embeddings and how to reproduce the error? Thanks.

[maccman]

@ocavue can you put this on your list.

[ocavue]

Here is a demo: https://github.com/issueset/voy-issue-50

Steps to run this demo:

# enable Node.js corepack to install pnpm.
$ corepack enable 

# clone the repo
$ git clone https://github.com/issueset/voy-issue-50
$ cd voy-issue-50

# install dependencies and run the app 
$ pnpm install
$ pnpm run dev 

Open http://localhost:5173/ and click the button, you will see the following error:

The core part of this demo is javascript/entry-browser.ts and data/value.json.

[DawChihLiou]

@ocavue thanks for the demo. I'm currently on vacation. Will look into it in a couple weeks.

[DawChihLiou]

The issue is caused by the memory allocation limit for WebAssembly. Browsers arbitrarily assign up to 4GB memory for wasm. In #48, I increased the bucket size to allow more data points in the index but it led to stack overflow during the deserialization process. Because the memory limit is a hard requirement for the browsers until memory64 arrives, I think it's the best to revert the bucket size. To index large amount of resources, I suggest to use multiple indexes to avoid the error in #48.

[DawChihLiou]

Fix released in v0.6.3.