search

kiyo-masui / bitshuffle

Filter for improving compression of typed binary data.
Other
220 stars 76 forks source link

Lock free decompression #139

Open jonwright opened 1 year ago

jonwright commented 1 year ago

As mentioned in issue #138, you can get a speed up for certain datasets on large CPU's by using a static openmp schedule. In this pull request, all threads read all offsets to find the compressed data and output pointers are computed.

Depending on whether load balancing is needed by other people, it might be better to use schedule(runtime) instead of schedule(static).

kiyo-masui commented 2 months ago

Just coming back to this after the person who maintained this package for the last few years has moved on to other things. This looks very cool. Could you update this branch with everything that's been added in the interim? That should trigger the continuous integration.

jonwright commented 2 months ago

So far I only clicked "update branch" on github - presumably it will need some more work before it will be ready.

kiyo-masui commented 2 months ago

Tests fail with a "bus error 10" which sounds like some sort of issue addressing memory.

jonwright commented 2 months ago

Sorry for that - the init was missed for non-openmp builds - it should be ready for you to look at now