Elasticlunr is a small, full-text search library for use in the Elixir environment. It indexes JSON documents and provides a friendly search interface to retrieve documents.
The changes contained in this PR allow the library to make use of ETS to store underlying indexed data. This improves performance and most importantly reduces the amount of data needed to be copied when the Elasticlunr.IndexManager.save/1 is called, as this is an obvious bottleneck.
Also, Task.async_stream/2 is now used when indexing documents because this allows us to analyze and process multiple documents concurrently.
TODO
[x] Decouple underlying index store to use ETS
[x] Adjust serialization/deserialization layer to work with these changes
[x] Run most indexing functions in an async stream
Overview
The changes contained in this PR allow the library to make use of ETS to store underlying indexed data. This improves performance and most importantly reduces the amount of data needed to be copied when the
Elasticlunr.IndexManager.save/1is called, as this is an obvious bottleneck.Also,
Task.async_stream/2is now used when indexing documents because this allows us to analyze and process multiple documents concurrently.TODO