Add Snappy support for BIG Endian machines such as s390x

airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java

Apache License 2.0

562 stars 111 forks source link

Add Snappy support for BIG Endian machines such as s390x #117

Closed kiwi1969 closed 1 year ago

kiwi1969 commented 3 years ago

Alternate classes provided for BIG ENDIAN Snappy support.

having separate means no IF statements everywhere checking for BIG/LITTLE endian conversions (ie performance), and means no impact to existing LITTLE_ENDIAN classes.

I tried initially to add IF statement into into SnappyCompressor and SnappyDecompressor mainline, but then this broke the Test scripts somehow. So instead chose the safe approach, of having separate main classes calling separate subclasses.

Users just need to use the Big Endian methods (ending in 'BE')in their programs if they are on a big Endian machine.

I Haven't exhaustively tested every flow, but seems to work for me, and is better than the nothing I had before.

kiwi1969 commented 3 years ago

I did a performance test, and the speed with both the BIG_ENDIAN and LITTLE_ENDIAN combined into same raw classes is not significant. So I combined and simplified the scope of changes.

kiwi1969 commented 3 years ago

Um no idea why this review has not happened yet. In the meantime, people needing big endian snappy should probably switch to snappy-java, as they accepted my Java code change, as well as provided a C++ method via JNI call.

dain commented 1 year ago

This project is focused on providing fast/efficient implementations of the compression algorithms, and that is currently not possible if we try to support different endiannesses. Additionally, we are not aware of any public CI infrastructure that supports big endian, so we could not test this change even if we wanted to suppor big endian. As we drop support for older JVMs we will evaluate new technology like VarHandles and MemorySegments, and those technonogies can transparently handle different endiannesses.