Closed ludwig-nc closed 4 years ago
dataRead should handle the byte order. But first, are your data sorted or unsorted ? I guess sorted because of 'time_3_3' but please confirm, 2 different parts of code. Assuming sorted and checking with MDFValidator for instance, is 'time_3_3' data type 4 (LE) or 5 (BE) ? Should be 5 as you have LE machine. Normally, read_double in dataRead module should swap the bytes simply using numpy with a .byteswap(). Maybe a flag wrongly calculated. What would help would be to dump the sorted_data_read() arguments in mdf4reader.py, read_channels_from_bytes() when processing 'time_3_3', I could potentially narrow down in dataRead() the bug if the bug does not yet come from mdf4reader.
Hi Aymeric,
Thanks a lot for looking into this issue. The File is a sorted MDF4 File. The data type of the time channel according to MDF Validator is 2=SIGNED_INTEL (for comparison the corresponding data channel is of type 0=UNSIGN_INTEL)
read_channels_from_bytes
is called repeatedly with the arguments:
channel_set = {'time_3_3', 'EngRPM'}
n_records=1000
dtype=None
channels_indexes=None
then sorted_data_read
gets called repeatedly twice (I assume the first time is for 'time_3_3' and the second time for 'EngRPM') with the following arguments:
print(self[chan].bit_count(info),
self[chan].signal_data_type(info),
self[chan].native_data_format(info),
n_records, self.CGrecordLength,
self[chan].bit_offset(info),
self[chan].pos_byte_beg(info),
self[chan].calc_bytes(info, aligned=False),
array_flag)
>>> 64 2 i8 1000 107 0 0 8 0
print(self[chan].bit_count(info),
self[chan].signal_data_type(info),
self[chan].native_data_format(info),
n_records, self.CGrecordLength,
self[chan].bit_offset(info),
self[chan].pos_byte_beg(info),
self[chan].calc_bytes(info, aligned=False),
array_flag)
>>> 14 0 u2 1000 107 0 23 2 0
It is weird to have signed int for a time signal.. But anyway, I think I found the obvious bug in dataRead, line 125 and 126, should not be 0,1 but 2,3 You can check in dev branch for the fix -> you will need to recompile dataRead
Many thanks, all is fine now. Are you planning on applying the same fix to the master branch?
Yes, I will issue new tag soon after some quality check
Python version
'3.7.4 (default, Jul 9 2019, 18:13:23) \n[Clang 10.0.1 (clang-1001.0.46.4)]'
Platform information
'Darwin-19.5.0-x86_64-i386-64bit' same problem exits on linux machine
Numpy version
'1.18.1'
mdfreader version
'4.1' dev branch
Description
unfortunately I am using proprietary data files and I cannot send you an example MD4-file, but hopefully I narrowed down my problem enough for you to have to have a look. The problem only exists when I am using the dataRead module and the
channel_list
argument. When reading in the whole MD4-file or when I am removing the dataRead.so everything is fine.The problem is as follows: When I am reading in a single channel of the file the data of the channel itself is fine but the corresponding master channel is read in with the wrong endianness. I am reading in the channel with the
convert_after_read=False
flag:when flipping the endianness of this I get the correct 100 Hz sampled time channel after applying the conversion factor of 1e-9: