meilisearch / grenad

Tools to sort, merge, write, and read immutable key-value pairs :tomato:
https://docs.rs/grenad
MIT License
25 stars 3 forks source link

Glidesort experiments #39

Open Kerollmops opened 1 year ago

Kerollmops commented 1 year ago

This PR exposes two cargo features to enable glidesort as the sort algorithm for the grenad Sorter:

Glidesort can also take the external buffer as a parameter which allows us to reduce the number of new allocations and speedup the Sorter again. The Sorter could allocate this buffer when created and keep it until dropped. Still, it could just take a slice of the available memory in the big buffer already allocated to store the keys and EntryBounds. It is a possible future work in this PR.

Kerollmops commented 1 year ago

Indeed, I will run the benchmark of milli and meilisearch and also run a real use-case of 40 million musics on a bare-metal instance just for fun 🤩