LaurentBlanc73 commented 3 months ago

Python version

3.9.2

Numpy version

1.26.1

mdfreader version

4.1

Description

Got the error: Big-endian buffer not supported on little-endian compiler when working with the generated dataframe. The last two lines of the following code evoked the error:

        df_new = mdf.return_pandas_dataframe(master)
        df_new.index = df_new.index.astype("datetime64[ms]")
        df_new = df_new[~df_new.index.duplicated()]
        df = pd.merge(df, df_new, left_index=True, right_index=True, how="outer")

The file type is .MDF

Resolution:

Added the following lines in line 1533 of mdfreader.py.

                if data.dtype.byteorder not in ['=', '|']:
                    data = data.byteswap().newbyteorder()

The if clause checks, if data is in the native byte order (or not applicable). If this is not the case, the second line gets executed which changes the byteorder.

It would be really useful if this would be implemented. Maybe there is another function for which it is also applicable (convert_to_pandas maybe?).

Sources:

https://tedboy.github.io/pandas/gotchas/gotchas10.html https://numpy.org/doc/stable/reference/generated/numpy.dtype.byteorder.html

LaurentBlanc73 commented 1 month ago

Hi @ratal, I really like the package. What do you think of my suggestion? It works perfectly for me :) I'm looking forward hearing from you

ratal commented 1 month ago

@LaurentBlanc73 Thanks for your proposal. Indeed, it seems that pandas do not like non native byte order. Could you please submit a Pull Request ?