Open SC426 opened 1 year ago
Hi. I think we have never tried C-Blosc2 with s390x arch before. Also, a cursory look into your output does not point to something immediately apparent to me. In case you find the issue, we would be happy to incorporate the fix. Thanks.
Hi @FrancescAlted Do you know if c-blosc2 has the capability to work on big-endian architectures? If so, could you give some pointers to where this is handled for other (supported) big-endian archs? I would be happy to submit a PR if we can get this working.
Hi @ELundby45 . Well, C-Blosc2 should be big-endian friendly, except when reading a frame from disk. This part starts at the comment line '// TODO: make this portable across different endianness` and was introduced in commit: https://github.com/Blosc/c-blosc2/commit/8b5ea6f92b9e42f681cba61756caea248045834d#diff-5de69465579e00aa53d6ff4e7fb060d8c08a03aa1721255fce9de3afbb27ba1a
The issue appears to be that the values in the chunk index are written in little-endian, but there is no protection for big-endian machines when reading them back. For more information on the chunk index, see https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst#chunks and search for chunk idx
. This can be fixed, but it requires some work. Let us know if you need more information.
I see something similar. The tests are very unhappy on s390x:
The following tests FAILED:
1 - test_plugin_test_ndlz (Failed)
2 - test_plugin_test_zfp_acc_float (Failed)
3 - test_plugin_test_zfp_prec_float (Failed)
4 - test_plugin_test_zfp_rate_float (Failed)
5 - test_plugin_test_zfp_rate_getitem (Failed)
6 - test_plugin_test_ndcell (Failed)
7 - test_plugin_ndmean_repart (Failed)
8 - test_plugin_ndmean_mean (Failed)
9 - test_plugin_bytedelta (Failed)
10 - test_b2nd_append (SEGFAULT)
11 - test_b2nd_copy (SEGFAULT)
12 - test_b2nd_delete (SEGFAULT)
13 - test_b2nd_full (SEGFAULT)
14 - test_b2nd_get_slice (SEGFAULT)
15 - test_b2nd_get_slice_buffer (SEGFAULT)
16 - test_b2nd_insert (SEGFAULT)
17 - test_b2nd_metalayers (Failed)
18 - test_b2nd_open_offset (SEGFAULT)
19 - test_b2nd_persistency (SEGFAULT)
20 - test_b2nd_resize (SEGFAULT)
21 - test_b2nd_roundtrip (SEGFAULT)
22 - test_b2nd_save (SEGFAULT)
23 - test_b2nd_serialize (SEGFAULT)
24 - test_b2nd_set_slice_buffer (SEGFAULT)
25 - test_b2nd_squeeze (SEGFAULT)
26 - test_b2nd_squeeze_index (SEGFAULT)
27 - test_b2nd_uninit (SEGFAULT)
28 - test_b2nd_zeros (SEGFAULT)
29 - generate_inputs_corpus (SEGFAULT)
302 - test_copy (Failed)
303 - test_delete_chunk (Failed)
306 - test_dict_schunk (Failed)
308 - test_fill_special (SEGFAULT)
309 - test_frame (Failed)
311 - test_frame_offset (SEGFAULT)
312 - test_get_slice_buffer (Failed)
713 - test_insert_chunk (Failed)
714 - test_lazychunk (Failed)
723 - test_reorder_offsets (Failed)
725 - test_schunk_frame (SEGFAULT)
726 - test_schunk_header (SEGFAULT)
727 - test_set_slice_buffer (Failed)
728 - test_sframe (Failed)
729 - test_sframe_lazychunk (Failed)
863 - test_small_chunks (Failed)
864 - test_udio (SEGFAULT)
865 - test_update_chunk (Failed)
866 - test_urcodecs (SEGFAULT)
868 - test_zero_runlen (Failed)
869 - test_compat_blosc-1.11.1-blosclz.cdata (Failed)
870 - test_compat_blosc-1.11.1-lz4.cdata (Failed)
871 - test_compat_blosc-1.11.1-lz4hc.cdata (Failed)
872 - test_compat_blosc-1.11.1-zlib.cdata (Failed)
873 - test_compat_blosc-1.11.1-zstd.cdata (Failed)
874 - test_compat_blosc-1.14.0-blosclz.cdata (Failed)
875 - test_compat_blosc-1.14.0-lz4.cdata (Failed)
876 - test_compat_blosc-1.14.0-lz4hc.cdata (Failed)
877 - test_compat_blosc-1.14.0-zlib.cdata (Failed)
878 - test_compat_blosc-1.14.0-zstd.cdata (Failed)
879 - test_compat_blosc-1.17.1-lz4-bitshuffle4-memcpy.cdata (Failed)
880 - test_compat_blosc-1.17.1-lz4-bitshuffle8-nomemcpy.cdata (Failed)
881 - test_compat_blosc-1.18.0-lz4-bitshuffle4-memcpy.cdata (Failed)
882 - test_compat_blosc-1.18.0-lz4-bitshuffle8-nomemcpy.cdata (Failed)
883 - test_compat_blosc-1.3.0-blosclz.cdata (Failed)
884 - test_compat_blosc-1.3.0-lz4.cdata (Failed)
885 - test_compat_blosc-1.3.0-lz4hc.cdata (Failed)
886 - test_compat_blosc-1.3.0-zlib.cdata (Failed)
887 - test_compat_blosc-1.7.0-blosclz.cdata (Failed)
888 - test_compat_blosc-1.7.0-lz4.cdata (Failed)
889 - test_compat_blosc-1.7.0-lz4hc.cdata (Failed)
890 - test_compat_blosc-1.7.0-zlib.cdata (Failed)
891 - test_compat_blosc-2.0.0-lz4-bitshuffle4-memcpy.cdata (Failed)
892 - test_compat_blosc-2.0.0-lz4-bitshuffle8-nomemcpy.cdata (Failed)
906 - test_bench_zero_runlen (Failed)
912 - test_example_frame_metalayers (SEGFAULT)
916 - test_example_frame_simple (SEGFAULT)
920 - test_example_frame_vlmetalayers (SEGFAULT)
921 - test_example_sframe_simple (Failed)
922 - test_example_frame_backed_schunk (Failed)
924 - test_example_frame_offset (SEGFAULT)
925 - test_example_frame_roundtrip (Subprocess aborted)
928 - test_example_frame_big (Failed)
929 - b2nd_example_empty_shape (Failed)
930 - b2nd_example_frame_generator (Failed)
931 - b2nd_example_oindex (Failed)
932 - b2nd_example_plainbuffer (Failed)
933 - b2nd_example_plugins_codecs (Failed)
934 - b2nd_example_plugins_filters (Failed)
935 - b2nd_example_print_meta (SEGFAULT)
936 - b2nd_example_serialize (Failed)
If needed I can provide access to a s390x system for development/debugging.
Same issue on debian/sid The full log is available at https://buildd.debian.org/status/fetch.php?pkg=c-blosc2&arch=s390x&ver=2.12.0%2Bds-1&stamp=1703958645&raw=0
+1 for avalentino
also fails with a newer version 2.13.1: https://buildd.debian.org/status/fetch.php?pkg=c-blosc2&arch=s390x&ver=2.13.1%2Bds-1&stamp=1706365370&raw=
as well as on Ubuntu with 2.13.1+ds-1ubuntu1: https://launchpadlibrarian.net/713964115/buildlog_ubuntu-noble-s390x.c-blosc2_2.13.1+ds-1ubuntu1_BUILDING.txt.gz
A friendly reminder that this is a known issue, and help would be appreciated to fix it. For details, see https://github.com/Blosc/c-blosc2/issues/467#issuecomment-1503692997.
I am trying to build a Conda package for c-blosc2. It is passing on all architectures except for IBM-Z (linux s390x), which is failing due to tests not passing. I have 2 questions:
Steps to reproduce: Run conda build on s390x
Expected behavior: Tests pass
Full build log: c-blosc2 s390x failure.log
GCC compiler linux-s390x Version 2.8.0