danielhrisca / asammdf

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files
GNU Lesser General Public License v3.0
612 stars 216 forks source link

Reading channels of boolean arrays fails #918

Closed victorjoh closed 8 months ago

victorjoh commented 9 months ago

Python version

'python=3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]' 'os=Linux-6.5.5-200.fc38.x86_64-x86_64-with-glibc2.37' 'numpy=1.26.0' 'asammdf=7.3.14'

Code

MDF version

4.10

Code snippet

from asammdf import MDF, Signal
import numpy as np

timestamps = np.array([0.1, 0.2, 0.3, 0.4, 0.5], dtype=np.float32)

samples = [np.ones((5, 2), dtype=np.uint8)]
types = [("boolean_array_channel", "(2, )<u1")]
record = np.core.records.fromarrays(samples, dtype=np.dtype(types))
boolean_array_channel = Signal(
    record,
    timestamps=timestamps,
    name="boolean_array_channel",
)

mdf4 = MDF(version="4.10")
mdf4.append(signals=[boolean_array_channel], comment="Created by Python")
# set bit count to 1 to indicate that each uint8 value is a boolean flag in boolean_array_channel
mdf4.groups[0].channels[1].bit_count = 1
signal = mdf4.select([('boolean_array_channel', 0, 1)])[0]

assert((record == signal.samples).all())

Traceback

Traceback (most recent call last):
  File "/home/victor/Sandbox/simon-karlsson-cde-2023-09-28/reproduce.py", line 19, in <module>
    signal = mdf4.select([('boolean_array_channel', 0, 1)])[0]
  File "/home/victor/anaconda3/envs/asammdf/lib/python3.9/site-packages/asammdf/mdf.py", line 3443, in select
    for idx, sigs in enumerate(
  File "/home/victor/anaconda3/envs/asammdf/lib/python3.9/site-packages/asammdf/blocks/mdf_v4.py", line 8277, in _yield_selected_signals
    signal = self.get(
  File "/home/victor/anaconda3/envs/asammdf/lib/python3.9/site-packages/asammdf/blocks/mdf_v4.py", line 6692, in get
    vals, timestamps, invalidation_bits, encoding = self._get_array(
  File "/home/victor/anaconda3/envs/asammdf/lib/python3.9/site-packages/asammdf/blocks/mdf_v4.py", line 7106, in _get_array
    vals = frombuffer(data_bytes, dtype=dtype_fmt)["vals"]
ValueError: buffer size must be a multiple of element size

Description

No mdf file is needed to reproduce this issue. The mdf is constructed in the code above to reproduce the problem. We create an mdf from scratch where we have a signal that is a channel array of boolean values. Since it is booleans we set the bit count to 1. When trying to read the signal from the mdf, we get an error.

I found this issue when dealing with an mdf generated from the tool Silver (T-2022.03-2) from QTronic. The mdf created above reproduces that problem I originally found when importing the mdf from Silver with asammdf.

victorjoh commented 9 months ago

asammdf is using bit_count / 8 to calculate the end offset when looking at the data_bytes (see traceback) asammdf-bit-count

instead I think asammdf should look at byte_offset_base (no division by 8) to get the offset asammdf-byte-offset-base

It makes sense when looking at the descriptions in MDF Validator, but it could also be that Silver has misinterpreted the MDF standard. I don't have a copy of the standard myself so I cannot validate what interpretation is correct.

It should be a simple fix in asammdf. I can make a PR for it.