Failing test on big-endian: TypeError: No conversion path for dtype: dtype('>U23')

[EdwardBetts]

When I run test tests on a machine with the s390x architecture the test_astropy_time_array fails with this exception:

TypeError: No conversion path for dtype: dtype('>U23')

This looks like an error caused by the s390x architecture being big-endian.

Here is the full output of the failing test.

$ python3 -mpytest --verbose -k test_astropy_time_array --no-cov
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0+repack -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/edward/hickle/hickle/.hypothesis/examples')
rootdir: /home/edward/hickle/hickle, configfile: tox.ini
plugins: doctestplus-0.12.0, arraydiff-0.5.0, openfiles-0.5.0, cov-3.0.0, mock-3.8.2, hypothesis-6.36.0, filter-subpackage-0.1.1, remotedata-0.3.3, astropy-header-0.2.1, astropy-0.10.0
collected 102 items / 101 deselected / 1 selected                                                                                                                                                                   

hickle/tests/test_06_load_astropy.py::test_astropy_time_array FAILED                                                                                                                                          [100%]

===================================================================================================== FAILURES ======================================================================================================
______________________________________________________________________________________________ test_astropy_time_array ______________________________________________________________________________________________

h5_data = <HDF5 group "/root_group" (2 members)>, compression_kwargs = {}

    def test_astropy_time_array(h5_data,compression_kwargs):
        """
        test proper storage and loading of astropy time representations
        """

        loop_counter = 0

        for times in ([58264, 58265, 58266], [[58264, 58265, 58266], [58264, 58265, 58266]]):
            t1 = Time(times, format='mjd', scale='utc')

            h_dataset, subitems = load_astropy.create_astropy_time(t1,h5_data, f'time_{loop_counter}',**compression_kwargs)
            assert isinstance(h_dataset,h5.Dataset) and not subitems and iter(subitems)
            assert h_dataset.attrs['format'] in( str(t1.format).encode('ascii'),str(t1.format))
            assert h_dataset.attrs['scale'] in ( str(t1.scale).encode('ascii'),str(t1.scale))
            assert h_dataset.attrs['np_dtype'] in( t1.value.dtype.str.encode('ascii'),t1.value.dtype.str)
            reloaded = load_astropy.load_astropy_time_dataset(h_dataset,b'astropy_time',t1.__class__)
            assert reloaded.value.shape == t1.value.shape
            assert reloaded.format == t1.format
            assert reloaded.scale == t1.scale
            for index in range(len(t1)):
                assert np.allclose(reloaded.value[index], t1.value[index])
            loop_counter += 1

        t_strings = ['1999-01-01T00:00:00.123456789', '2010-01-01T00:00:00']

        # Check that 2D time arrays work as well (github issue #162)
        for times in (t_strings, [t_strings, t_strings]):
            t1 = Time(times, format='isot', scale='utc')

>           h_dataset,subitems = load_astropy.create_astropy_time(t1,h5_data,f'time_{loop_counter}',**compression_kwargs)

/home/edward/hickle/hickle/hickle/tests/test_06_load_astropy.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/edward/hickle/hickle/hickle/loaders/load_astropy.py:134: in create_astropy_time
    d = h_group.create_dataset(
/usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/group.py:161: in create_dataset
    dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
/usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/dataset.py:88: in make_new_dset
    tid = h5t.py_create(dtype, logical=1)
h5py/_debian_h5py_serial/h5t.pyx:1663: in h5py._debian_h5py_serial.h5t.py_create
    ???
h5py/_debian_h5py_serial/h5t.pyx:1687: in h5py._debian_h5py_serial.h5t.py_create
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: No conversion path for dtype: dtype('>U23')

h5py/_debian_h5py_serial/h5t.pyx:1753: TypeError
============================================================================================== short test summary info ==============================================================================================
FAILED hickle/tests/test_06_load_astropy.py::test_astropy_time_array - TypeError: No conversion path for dtype: dtype('>U23')
========================================================================================= 1 failed, 101 deselected in 0.86s =========================================================================================
$

[EdwardBetts]

git bisect says 724ea2957dd0cd7605752c423df3046b0865e113 is the commit that caused the test to start failing on s390x

[telegraphic]

😱 Endianess is always fun. Thanks for tracking down commit. Tagging help-wanted as I'm low on cycles at the moment, but will eventually get to this

[telegraphic]

Hi @EdwardBetts, not sure if this has fixed the issue, but I hope so! The test suite doesn't currently run on s390x, although that may be possible via github actions

[telegraphic]

It seems that unfortunately there is no in-built runner for s390x (as far as I can tell, and as you may already know!)

[EdwardBetts]

Here's the test failing for version 5.0.2. Hopefully the tests will start passing with the next release.

See https://qa.debian.org/excuses.php?package=hickle and https://ci.debian.net/packages/h/hickle/testing/s390x/44506354/

[EdwardBetts]

This post about using docker and github actions to run tests on the S390 architecture might be helpful. https://til.simonwillison.net/docker/emulate-s390x-with-qemu

[telegraphic]

Hmm, it's failing on

 45s tests/test_01_hickle_helpers.py::test_H5NodeFilterProxy FAILED           [  3%]

which was fixed by @mr-c --for Debian -- but I think I un-fised it when merging.

The v5.0.3 release should fix that particular issue (which shouldn't be S390 specific).

[mr-c]

The tests are passing on s390x with release 5.0.3, thank you!

https://ci.debian.net/packages/h/hickle/testing/s390x/44538580/