Open danking opened 1 month ago
There should be array validator that you could implement for an array then it would happen once on construction. We minimally try to deserialize metadata but that doesn't happen all the time and doesn't cover the case here. Then we could use the View types
Both
RoaringIntArray
andRoaringBoolArray
useBitmap::deserialize
which copies the source bytes. There existsBitmapView::deserialize
but it is unsafe. We either should verify the correctness of the source bytes and use the unsafe method or we should find another way to deserialize a roaring array without copying.In my experience working with the PBI datasets, for example CMSprovider, copying roaring arrays can be ~10% of total runtime when decompressing.