danielhrisca / asammdf

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files
GNU Lesser General Public License v3.0
612 stars 216 forks source link

Can DataBytes Channel missing when exporting as parquet file #963

Open suiluj opened 6 months ago

suiluj commented 6 months ago
# Python version
('python=3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 '
 'bit (AMD64)]')  

'os=Windows-10-10.0.19044-SP0'  

'numpy=1.26.1'

Code

MDF version

I used an MDF4 file

Code snippet

# https://asammdf.readthedocs.io/en/latest/api.html#asammdf.mdf.MDF.export
mdf_object_to_export.export(
    fmt="parquet", keep_arrays=True, raw=True, filename=export_file_name_path, **other_kwargs
)
# (tried different combinations of raw and keep_arrays arguments)

Traceback

no traceback/error just missing column in parquet file.

Description

I am trying to export a CAN trace as parquet and just want to export the whole mdf_object via .export()

Everything works fine but the collumn/channel DataBytes is missing in the parquet file.

image

In the asammdf GUI program I can open the CAN trace view and the column shows the DataBytes data correctly as byte array in hex.

When I export as .csv file the column is export like this (byte array in decimal numbers):

# one example of one row of the DataBytes column
[240 160   0   10  20   30   40  50]

It seems that the parquet export just ignores byte arraycolumns.

I already tried the argument raw=True and keep_arrays=True but I did not find the column in the result either.

Some other issues which describe similar problems:

https://github.com/danielhrisca/asammdf/issues/918

https://github.com/danielhrisca/asammdf/issues/186

https://github.com/danielhrisca/asammdf/issues/299

Does somebody have a solution? I really would like to export to parquet instead of csv because then I can query the data faster.