search

apache / fury

A blazingly fast multi-language serialization framework powered by JIT and zero-copy.
https://fury.apache.org/
Apache License 2.0
3.06k stars 239 forks source link

[Java] primitive array compression by SIMD #110

Open chaokunyang opened 1 year ago

chaokunyang commented 1 year ago

Is your feature request related to a problem? Please describe. Currently primitive array are serialized by copy the data buffer directly using sun.misc.Unsafe, this is super fast. Buf if all values in array are small, compression may reduce the data dramatically.

Compression needs to iterate the array at least once, which may be slow.

Describe the solution you'd like

Additional context

108

heliang666s commented 6 months ago

Hello, can you describe the solution in more detail？

chaokunyang commented 6 months ago

Hello, can you describe the solution in more detail？

Hi @heliang666s , thanks for the interest about this issue.

We may need to use SIMD api to detect whether all values in array are small value. If all values are small or first n values are small, we should use SIMD to cast it into smaller value into target buffer directly .

heliang666s commented 6 months ago

ok,I will try it.Please assign me thanks~