Open halgari opened 1 week ago
In NX 2.0 (my enhanced Rust port), I'm planning to leverage xxHash3.
I would love to use GxHash, but there's some catches:
The combination of the two means that if, for example RISC-V started to infiltrate the PC market eventually, we wouldn't be able to support it, unless we contribute an implementation or fallback ourselves. And that would be problematic, because we'd be using a version of GxHash which would be years out of date at that point. Because we're assuming the algorithm to be immutable (not changeable) that rules GxHash out. Same with t1ha0
and other similar algorithms in class.
Misc Note: As for hardware availability, GxHash should be supported on all x86 CPUs since 2015 (Skylake), and on some machines between 2010-2015 (depending on class).
Data Model Solidification
As part of our upcoming "Data Model migration support" issues, there is a small list of things that are way easier to change now than in the future. For these issues we should do some investigation into our needs and not just decided if we want to change them, but if we will ever want to change them. We'll have the ability to make significat changes to the datamodel in the future, but fundamental changes like the ones listed below will be hard if not impossible to change.
List of "hard to change in the future" features
Another option is GxHash, it's only 2 years old and exists only in Rust it appears but is about 4x faster than xxHash3 although it requires certain intrinsics to work properlyIndexSegment
objects and to track segments of the index data separately from Rocks's caches. This means that all loaded data is essentially duplicated in memory. Due to the way C interop works in C# we cant cleanly use RocksDB iterators directly and have IndexSegments be RocksDB iterator ranges.[ ] Window Serialization Data
Context
object in addition to the page type in the window code...I'll write up a different issue for this