Opening of CANape measurement raises numpy exception

[rolfub]

Python version

'3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]'

Platform information

'Windows-10-10.0.16299-SP0'

Numpy version

'1.16.4'

mdfreader version

'3.2'

mdf file version

MDF 4.10 MCD15.03

CANape

tool_vendor>Vector Informatik GmbH

Description

Hello Aymeric,

I have a problem with a CANape measurement. Numpy raises an exception. Could you please take a look at it? Perhaps you can prevent the exception.

Here is the python traceback: C:\Users\Rolf\Desktop>C:\Python37\python.exe MDF_read.py C:\Python37\lib\site-packages\mdfreader\mdfinfo4.py:604: UserWarning: Could not parse CG block names tag warn('Could not parse CG block names tag') C:\Python37\lib\site-packages\mdfreader\mdfinfo4.py:604: UserWarning: Could not parse CG block names tag warn('Could not parse CG block names tag') C:\Python37\lib\site-packages\mdfreader\mdfinfo4.py:604: UserWarning: Could not parse CG block names tag warn('Could not parse CG block names tag') Traceback (most recent call last): File "MDF_read.py", line 70, in oMDF = mdfreader.Mdf( "C:\Users\Rolf\Desktop\CANape_Measurement.MF4" ) File "C:\Python37\lib\site-packages\mdfreader\mdf.py", line 160, in init metadata=metadata) File "C:\Python37\lib\site-packages\mdfreader\mdfreader.py", line 413, in read convert_after_read, compression, metadata) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 1309, in read4 buf.read(channel_set, info, self.fileName) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 366, in read record[cn].name, temp) File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 787, in rec_append_fields asrecarray=True, usemask=False) File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 720, in append_fields data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)] File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 720, in data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)] File "C:\Python37\lib\site-packages\numpy\core_internal.py", line 494, in _view_is_safe raise TypeError("Cannot change data-type for object array.") TypeError: Cannot change data-type for object array.

Here is self[recordID]['data'] from line 366: self[recordID]['data'] = [ (9.35309800e+06, 29, 4, 1138, 17, b'\x00\x00\x00\x00\x00\x00\x00\x00') ...

Thank you!

Your sincerely,

Rolf

[ratal]

Hi Rolf, VLSD is not so used and tested. I will try to investigate but as usual, if you have possibility to share file, it will be easier. Otherwise, can you also show the related CG and CN blocks with MDFValidator ? + content of temp when failing (line 336 of mdf4reader)

[rolfub]

Hello Aymeric, thank you for investigating! I know it, but I must not give you the mdf file. It is a company secret. If have created a file with debug information and send it to your e-mail adress. I hope it will help you. Have a good weekend! Rolf

P.S. Record.str() (line 621, mdfreader4.py): "chan.type" is an "int", so there will be a TypeError exception when you call join()

[ratal]

According to the mdfreader record structure, there is no VLSD channel in the CG Block. But record.VLSD is not empty. Probably the record contains bus data which DataBytes has variable length, complicated. Can you confirm what CN block has channel 14 DataBytes from bus FLX_Frame (and its cn_type) and cg block cg_flags ? Only one channel group in the data group (sorted data) ? So far mdfreader considers this a sorted datagroup but if there is VLSD, it shall be considered as unsorted.

[rolfub]

Hello Aymeric, sorry for the late answer. I have send you debug information again. I hope it helps. Yours sincerely, Rolf

[ratal]

Hi, Sorry for the delay. I have difficulties here. From the CN and CG blocks, Frame.DataBytes is Variable Length Signal Data of type byte array. Everything seems to go fine until in Data.read(), there is rec_append_fields that cannot handle recarray of type object array. I would say this is a numpy bug but can you also show what is temp ?

[rolfub]

Hello Aymeric, I am also sorry for the delay. I was on vacation the last weeks. I will send you debug information about temp. In Frame.Databytes should be data from the CAN bus tracing. Have a good weekend! Rolf

[ratal]

Hi Rolf, I have no file with variablelength bytearray to test with. However, converting a list of bytearrays with variable length into an array results in an array with object dtype, that rec_append_fields has difficulties with, so _equalize_byte_length() function is there for that. But I noticed for some reason that line 247 buf = _equalize_byte_length(buf)in _read_sd_block() was commented -> Can you try uncommenting it ?

[rolfub]

Hi Ayemric,

sorry, I must not give you a test file. :-( I uncommented "buf = _equalize_byte_length(buf)" in line 246 and got the following exception:

oMDF = mdfreader.Mdf( "C:\\Users\\rolf\\Desktop\\test.MF4" )

File "C:\Python37\lib\site-packages\mdfreader\mdf.py", line 160, in init metadata=metadata) File "C:\Python37\lib\site-packages\mdfreader\mdfreader.py", line 413, in read convert_after_read, compression, metadata) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 1310, in read4 buf.read(channel_set, info, self.fileName) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 363, in read temp = temp.load(record, info, name_list=channel_set, sorted_flag=True, vlsd=True) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 443, in load n_records=None, sorted_flag=sorted_flag, vlsd=vlsd) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 116, in _data_block parent_block['length'] - 24) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 246, in _read_sd_block buf = _equalize_byte_length(buf) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 278, in _equalize_byte_length max_len = len(max(buf, key=len)) TypeError: object of type 'NoneType' has no len()

1. while loop cycle: buf = [bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00v\xd4\xff\x00\x00\x00\x08\x01\x00\xe0?\xff\xf9\xcf\x7f\x00\x000')]

2. while loop cycle: buf = [None, bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x04\x01\x00\xe8\x0b\x19\x00\x00\x04')]

Yours sincerely,

Rolf

[ratal]

Please try with : buf[i] = bytearray(element).rjust(max_len, b'\x00')in the next for loop

[rolfub]

Hello Aymeric,

do you mean to change line 280? (-> for loop) But the exception is in line 278.

Yours sincerely,

Rolf

[ratal]

You are right, I anticipated this bug then :) But I do not understand the while loops 1&2 you mention. The bytearrays are different. Is it in _read_sd_block() ? From where is coming this None ? Buf should contain only bytearrays

[rolfub]

Hello Aymeric, I think the "None" was my fault because I only deleted the # before "_equalize_byte_length(buf)" and didn't look after the indent. Now it looks like this:

240 else: # byte arrays or mime types 241 while pointer < sd_block_length: 242 VLSDLen = structunpack('I', sd_block[pointer:pointer + 4])[0] # length of data 243 pointer += 4 244 buf.append(sd_block[pointer:pointer + VLSDLen]) 245 pointer += VLSDLen 246 buf = _equalize_byte_length(buf)

and it works until line 366 then comes the following exception:

oMDF = mdfreader.Mdf( "C:\\Users\\rol\\Desktop\\test.MF4" )

File "C:\Python37\lib\site-packages\mdfreader\mdf.py", line 160, in init metadata=metadata) File "C:\Python37\lib\site-packages\mdfreader\mdfreader.py", line 413, in read convert_after_read, compression, metadata) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 1310, in read4 buf.read(channel_set, info, self.fileName) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 366, in read record[cn].name, temp) File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 787, in rec_append_fields asrecarray=True, usemask=False) File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 744, in append_fields output = recursive_fill_fields(data, output) File "C:\Python37\lib\site-packages\numpy\lib\recfunctions.py", line 76, in recursive_fill_fields output[field][:len(current)] = current File "C:\Python37\lib\site-packages\numpy\ma\core.py", line 3329, in setitem _data[indx] = dval ValueError: could not broadcast input array from shape (5486,62) into shape (5486)

The Exception is different to the first one and temp is also different: temp = [[ 0 ... 48] ... [ 0 ... 16]]

Yours sincerely,

Rolf

[ratal]

ma is a masked array, meaning you probably have invalid bits.. Except that the error is too deep into numpy for me to follow, sorry : but can you check shapes for self[recordID]['data'] and temp for this exception ? Problematic channel could be different from previous one ? (same cn, datagroup # ?) because temp content do not seem to be like before (numeric instead of byte) but something could have been newly wrong at data reading

[rolfub]

Hello Aymeric, it is the same cn and data. With the change in line 280 from "buf[i] = bytearray(element).extend(b'0' * (max_len - len(element)))" to "buf[i] = bytearray(element).rjust(max_len, b'\x00')" comes the difference. Without change temp = [ None, .. None ]. I send you the debug data. Have a good weekend! Rolf

[ratal]

First issue, temp is 5486 long while it should be 21085 Then the content of temp is different from previous time. [0 0 0 0 ... 0 0] a list of int (dimension 62) while last time it was bytearray. _equalize_byte_length returns a list of bytearrays but when applying in _read_sd_block the array(buf), you obtain [0 0 0 0 ... 0 0] So can you modify (but it is probably not solving the size issue) _equalize_byte_length

    max_len = len(max(buf, key=len))
    output = bytearray()
    for i, element in enumerate(buf):  # resize string to same length, numpy constrain
        output += bytearray(element).rjust(max_len,  b'\x00')
    return buf, max_len

and _read_sd_block

        buf = array(_equalize_string_length(buf))
    else:  # byte arrays or mime types
        while pointer < sd_block_length:
            VLSDLen = structunpack('I', sd_block[pointer:pointer + 4])[0]  # length of data
            pointer += 4
            buf.append(sd_block[pointer:pointer + VLSDLen])
            pointer += VLSDLen
        buf, max_len = _equalize_byte_length(buf)
        buf = frombuffer(buf, dtype='V{}'.format(max_len))
    return buf

and add from numpy import frombuffer at beginnig of file

[rolfub]

Hello Aymeric,

i have made the changes as you said. Now there seems to be a type problem? Here is the traceback: File "MDF_read.py", line 71, in oMDF = mdfreader.Mdf( "C:\Users\rolf\Desktop\Test.MF4" ) File "C:\Python37\lib\site-packages\mdfreader\mdf.py", line 160, in init metadata=metadata) File "C:\Python37\lib\site-packages\mdfreader\mdfreader.py", line 413, in read convert_after_read, compression, metadata) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 1320, in read4 buf.read(channel_set, info, self.fileName) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 365, in read temp = temp.load(record, info, name_list=channel_set, sorted_flag=True, vlsd=True) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 453, in load n_records=None, sorted_flag=sorted_flag, vlsd=vlsd) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 117, in _data_block parent_block['length'] - 24) File "C:\Python37\lib\site-packages\mdfreader\mdf4reader.py", line 248, in _read_sd_block buf = frombuffer(buf, dtype='V{}'.format(max_len)) AttributeError: 'list' object has no attribute 'buffer'

The import for "frombuffer" was already in line 41. That should not be the problem. I can't say if there is still a problem with the size because of the exception.

Yours sincerely,

Rolf

[ratal]

Can you try to replace: output += bytearray(element).rjust(max_len, b'\x00') into output.extend(bytearray(element).rjust(max_len, b'\x00')) buf should not be a list but one big bytearray

[rolfub]

Hello Aymeric, I have changed it and the result was the same. So it was not the problem. I have not seen it yesterday but today. "buf" is not changed in _equalize_byte_length().

For testing I changed the return value of _equalize_byte_length() from "buf" to "output":

max_len = len(max(buf, key=len))
output  = bytearray()
for i, element in enumerate(buf):  # resize string to same length, numpy constrain
    output.extend( bytearray(element).rjust(max_len, b'\x00') )
return output, max_len

It works now without exception. The size problem seems to be solved too. Please take a look if you want to leave it like that? Thank you for your great support!

Yours sincerely,

Rolf

[ratal]

Hi Obviously, returning buf in _equalize_byte_length was meaningless ! :) Stupid mistake. Thanks for the feedback, closing the issue and pushing the fix !