search

aschampion / rust-n5

Rust implementation of the N5 "Not HDF5" tensor file system format
Apache License 2.0
15 stars 3 forks source link

Switch LZ4 from frame mode to block mode #27

Closed aschampion closed 5 years ago

aschampion commented 5 years ago

Java N5 uses block mode. Block mode was not supported by the rust lz4 crate when this implementation was created, but it now is supported.

aschampion commented 5 years ago

Closing as invalid since the Java lz4 spec block includes a frame descriptor with the magic bytes (0x184D2204).

aschampion commented 5 years ago

Note that the actual difference between Rust N5 and Java N5 here is that Rust N5 by default uses linked LZ4 blocks (data frame blocks unrelated to N5 blocks), while Java N5 uses independent LZ4 blocks. See the LZ4 spec.