danielhrisca / asammdf

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files
GNU Lesser General Public License v3.0
655 stars 226 forks source link

array-shape mismatch error if a referenced signal axis is shorter than the signal itself #1062

Open rjwolke opened 3 months ago

rjwolke commented 3 months ago

Python version

'os=Windows-10-10.0.19045-SP0'
'numpy=1.26.4'
'asammdf=7.4.2'

Code

MDF version

4.10

Code snippet

  m = asammdf.MDF("Short_Referenced_Axis.mf4")
  group, index = m.whereis("TVRSPDIVFBDLVIHKIQMVB")[0]
  m.get(group=group, index=index)

Traceback

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "C:\Work\40_Dev\asammdf\.venv\Lib\site-packages\asammdf\blocks\mdf_v4.py", line 6670, in get
      vals, timestamps, invalidation_bits, encoding = self._get_array(
                                                      ^^^^^^^^^^^^^^^^
    File "C:\Work\40_Dev\asammdf\.venv\Lib\site-packages\asammdf\blocks\mdf_v4.py", line 7226, in _get_array
      vals = fromarrays(arrays, dtype(types))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Work\40_Dev\asammdf\.venv\Lib\site-packages\numpy\core\records.py", line 676, in fromarrays
      raise ValueError(f'array-shape mismatch in array {k} ("{name}")')

Description

Short_Referenced_Axis.zip

The channel "TVRSPDIVFBDLVIHKIQMVB" is a curve signal that is referencing the channel "BOBNKYCWZQJVUAVHLDJHC" as its axis. When the value of TVRSPDIVFBDLVIHKIQMVB is changed during measurement, ETAS INCA will create an entry in the measurement for that channel, but not for the referenced channel. This causes the channel lengths to differ (the axis has one less entry), ultimately causing the above exception when accessing the signal in most ways (e.g. get, to_dataframe, merge).

The reason for not updating the axis is that mutliple signals may in theory reference that same axis channel, which would then require updating all channels referencing that axis whenever any one of them gets an entry.

Workaround

The problem can be worked around by replacing the axis reference with a reference to an interpolated copy of the axis. But since that will modify the original measurement file, it would be preferable not to do this.

  for dep in m.groups[group].channel_dependencies:
      if dep is None:
          continue

      for cab in dep:
          if not cab.axis_channels:
              continue

          # Get timestamps for the signal group
          timestamps = m.get_master(group)

          # Iterate over axis channels
          for i, (depgrp, depidx) in enumerate(cab.axis_channels):
              axis= m.get(group=depgrp, index=depidx)
              # Check if we need to fix the length
              if len(axis.timestamps) < len(timestamps):
                  m.append(
                      axis.interp(timestamps, 0, 0), 
                      comment = "", 
                      acq_name = "", 
                      acq_source = axis.source
                  )

                  # Replace reference with newly added signal group/index
                  cab.axis_channels[i] = m.whereis(axis.name)[-1]