BurntSushi
commented
3 years ago Do you think the library can be split in a index build part, and a query part, with only the latter being compiled to WASM?
The library almost certainly can be split. The question is whether it's worth doing and whether it would actually solve your problem. Here are my thoughts:
- My guess is that you're asking to make the "building" phase a feature gate, thus giving you the ability to build the library without it. This seems like a tractable amount of work and the maintenance burden is probably not too bad, but I'd want a really solid motivation for it. Because it does litter the code with a bunch of conditional compilation gates and that in general is pretty annoying to deal with.
- The "building" phase is really not a lot of code. Not even close to a majority, for example. So if the library is currently 30MB, removing the building phase of FSTs is unlikely to make it dramatically smaller. My guess is that you would prefer it to be closer to 1MB (or even less). It's unclear to me why you think removing the building phase specifically would get the compiled artifact to a size that makes what you want to do possible. Have you done any related experiments that gives you confidence that your suggestion will actually solve your problem?
- 30MB sounds like a lot. How are you getting that measurement? Are you sure you aren't, like, compiling the library in debug mode? Or in release mode with debug symbols enabled?
If the library really is 30MB as a WASM compiled artifact, then you'd probably need to do a lot of trimming beyond just "separate out the build phase." But to be honest, I have a really hard time believing 30MB to be correct. This library just isn't that big.
As an example of what I mean, I can mention here https://github.com/jameslittle230/stork, written in Rust, too, but which is not able to query as your library does.
To be honest, I'd expect Stork to do a lot better. An FST is itself not sufficient to build an IR system like Stork. An FST is merely a (possible) component of such a system. With that said, I've not done any investigation into how Stork works.
claudius108
Hi,
I tested your library by using the command line, in order to query for the words in a dictionary, and results are very fine, as to the query time, the possibility to search by using wildcards, and to the size of the index.
I would like to compile the library to WASM, in order to use it in browser (as a search engine for static sites), but its size is currently too large (30MB). Do you think the library can be split in a index build part, and a query part, with only the latter being compiled to WASM?
As an example of what I mean, I can mention here https://github.com/jameslittle230/stork, written in Rust, too, but which is not able to query as your library does.
Thank you, Claudius