Error opening mdf file from CANoe (mdf3)

[rolfub]

Hello,

I wanted to read a mdf file from CANoe (mdf version 3.x).

Code: import mdfreader file = mdfreader.mdf("CANOE.MDF")

But it failed with the following python traceback: Traceback (most recent call last): File "C:\Users\XXX\Desktop\test.py", line 16, in file = mdfreader.mdf("CANOE.MDF") File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf.py", line 114, in init self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdfreader.py", line 360, in read self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 825, in read3 buf.read(channelSet) # reads datablock potentially containing several channel groups File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 651, in read data = self.loadUnSorted(nameList=channelSet) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 696, in loadUnSorted for recordID in record: TypeError: 'type' object is not iterable

Could you please help me with this problem.

Rolf

[ratal]

reading unsorted mdf3 files is new feature (issue #42 ) but I could not test it as I never had a test file. If you could share a non sensitive file from your side, it would help finalising its development. Anyway, I will have a look.

[ratal]

I did few modification to latest commit to solve this issue, please try, probably hitting another.

[rolfub]

Hello,

sorry for the late answer. I had to clarify if I can send you a mdf file for developing.

Here is one file in mdf version 3.3 ("CANoe3.mdf"). It is created with CANoe.

I hope that the file would be a help for you.

I could check your last changes (thank you for them!) but I think that you can make it better with "CANoe3.mdf". :-)

Rolf

P.S. I have send the file to "aymeric.rateau@gmail.com".

[ratal]

Thanks for the file! Please have a try to latest commit, should be working.

[rolfub]

Hello,

with "CANoe3.mdf" it works. :-) But with "CANOE.mdf" it doesn't work. :-( Unfortunately I am not allowed to send this file to you. I hope that you can help me anyway.

Here is the python traceback:

Traceback (most recent call last): File "C:\Users\XXX\Desktop\Test.py", line 128, in file = mdfreader.mdf( "CANOE.mdf" ) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdf.py", line 114, in init self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRe ad, filterChannelNames=filterChannelNames) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdfreader.py", line 360, in read self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, fi lterChannelNames=False) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdf3reader.py", line 826, in read3 buf.read(channelSet) # reads datablock potentially containing several channe l groups File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdf3reader.py", line 652, in read data = self.loadUnSorted(nameList=channelSet) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdf3reader.py", line 703, in loadUnSorted temp = self[recordID]['record'].readRecordBuf(stream[position:position + sel f[recordID]['record'].CGrecordLength + 1], nameList) File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\ mdf3reader.py", line 591, in readRecordBuf temp[Channel.recAttributeName] = Channel.CFormat.unpack(buf[Channel.posByteB eg:Channel.posByteEnd])[0] struct.error: unpack requires a string argument of length 4

Thank you for your efforts!

Rolf

[ratal]

Bit difficult like this to understand what is wrong. Most probably a certain data structure is not well understood. A snapshot of mdfvalidator could help.

[rolfub]

Hello,

a snapshot of mdfvalidator? I have installed MDF Validator from Vector. :-) Which information will help you?

Rolf

[dapperfu]

I'm getting the same error on an MDF3 file from CANape.

[ratal]

I would like to see if there woudl be exotic data structure. I think you can export the files structure into a xml ? Otherwise a few screenshot of the tree maybe.

[dapperfu]

Do you know where that'd be? I think I only have access to CANape, I dug around the menus but didn't see any XML options.

Additionally if you gave me a hint for what to look for I might be able to find it.

[rolfub]

Hello,

here are the xml files with the structure of CANoe3.mdf (which I have send you) and CANoe.mdf.

I hope this is a help for you.

Rolf

P.S. The files are in the E-Mail.

[ratal]

A hint: introduce a print(Channel) before line 591 where you have the issue. Then compare the output with the channel definition that is giving you mdfvalidator.

[dapperfu]

I just forked and am getting a different error on a different line:

AttributeError: 'record' object has no attribute 'recAttributeName'
> c:\projects\mdfpython-64bit-3.6\mdfreader\mdfreader\mdf3reader.py(699)loadUnSorted()
-> for channelName in self[recordID]['record'].recAttributeName:

Oddly enough just opening the file in CANape 'fixes' it. CANape must sort the channels.

The good file uses loadSorted while the bad file uses loadUnsorted, which is where the error occurs.

These are the attributes that the record has at the place where the error occurs.

dir(self[recordID]['record'])
['CGrecordLength', '__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'addChannel', 'append', 'byte_aligned', 'channelGroup', 'channelNames', 'clear', 'copy', 'count', 'dataBlockLength', 'dataGroup', 'dataRecordName', 'extend', 'hiddenBytes', 'index', 'insert', 'loadInfo', 'master', 'numberOfRecords', 'numpyDataRecordFormat', 'pop', 'readRecordBuf', 'readSortedRecord', 'recordID', 'recordIDnumber', 'recordLength', 'recordToChannelMatching', 'remove', 'reverse', 'sort']

[ratal]

Jed-frey, correct attribute call should be dataRecordName

[ratal]

Hi rolfub, I did not receive any attachment. Can you instead send an email with your attachment at aymeric.rateau@gmail.com ?

[rolfub]

Hello ratal,

I have send you the files by e-mail.

Rolf

[ratal]

Hi rofub, My bad, sending this xml was not enough, I finally could extract very few information, only the fact your file contains many many channel groups, an unsorted file. It does not help me. I will have to guide you debugging: Can you introduce a

print(Channel)

before line 591 where you have the issue. Then compare the output with the channel definition that is giving you mdfvalidator -> any difference will cause this kind of error

[ratal]

Hi Rolfub I identified issues looking at your sent file in details. It appeared there was in each channel group same time channel 't'. I added new code to deal with it properly. Please have check, it could have solved your issue.

[rolfub]

Hello ratal,

sorry for the late answer but I don't see the problem. The only difference to the previous channel groups is the record data format.

Channel group 68 has the record data format '<u4' (Data type = 0, Bit count = 19) for the first time. Could it possibly be the 3 byte problem? You expect 4 bytes and get 3 bytes.

I have used the version with the newest change that you have mentioned and I have the same problem as before.

Rolf

[ratal]

Hi Rolfub, This kind of case should be handled by the recordToChannelMatching attribute of record class. It will inform if several channels are embedded into one temporary channel that is byte_aligned (property of record class), meaning channels bits are spread within bytes of 1, 2, 4 or 8. Later embedded channels are extracted. Could you print the record and check if it is detected byte_aligned and hiddenBytes and cross check with MDFValidator ? If it is not the case, the current mdf3.x code is not ready yet for this (mdf4.x should be). Please update to latest commit, it should show if it is the case. By the way, the file you sent me, is it same structure as the one you are having problems ?

[rolfub]

Hello ratal,

I have inserted some print() calls in line 605 in mdfreader3.py. It seems to be the three byte problem? What do you think?

This are the results from chanel group 68, in which the problem exists: print( Channel.CFormat.format ) --> <I print( Channel.posByteBeg ) --> 9 print( Channel.posByteEnd ) --> 13 print( len(buf[Channel.posByteBeg:Channel.posByteEnd]) ) --> 3 for i in xrange(Channel.posByteBeg, Channel.posByteEnd): --> , , , => IndexError: out of range print( buf[i] + "," )

From the view of the index everything is fine (--> 9 to 13). You calculate "posByteEnd" in mdf._bits_to_bytes() to four bytes (line 318 in mdfreader3.py). But if you ask for the length of the string or print the string, you get only three characters (one too little for unpack).

The file I have send you should have a similar structure. It is from an old project but we use the same software tests.

Rolf

[ratal]

Your old project might be recording different channels or CAN buses compared to your problematic file ? Can you record a dummy file similar to your concerned one for steady/null conditions ? Like this confidentiality should be ok but structure will be similar. Yes 3bytes channel could be the issue. This kind of case should be detected by setting byte_aligned attribute of record class to False; please check it. If this detection is effective for unsorted data, I added in last commit a new method to record class to read data called readRecordBits (still first shot, most probably buggy)

[rolfub]

Hello ratal,

I have inserted "print( self.byte_aligned )" in mdf3reader.py (line 603). I get 68 times "True" before the exception is thrown. Could be the Problem?

Rolf

[ratal]

Hi Rolfub, Trying to investigate more deeply your issue: Byte _aligned will indicate if the byte are shifted by some additionnal bits --> need to know also posBitBeg and posBitEnd Could you also check hiddenBytes ? Also recordToChannelMatching ?

[rolfub]

Hello ratal,

here are the values from my latest debug session:

Record.byte_aligned = True Record.hiddenBytes = False Record.recordToChannelMatching = {'CAN_Matrix_V10VIN_01VIN_16': 'CAN_Matrix_V10VIN_01VIN_16', 't_67': 't_67'}

Channel.name = t_67 Channel.posBitBeg = 8 Channel.posBitEnd = 72

Channel.name = CAN_Matrix_V1.0::VIN_01::VIN_16 Channel.posBitBeg = 72 Channel.posBitEnd = 91

I hope I could help you.

Rolf

[ratal]

Dear Rolfub, In your group, it seems there are only 2 channels but starting from bit position 8, weird. Anyway, this is compatible with specification and I notice hiddenBytes should have been activated --> I will investigate why it is not. Aymeric

[ratal]

hiddenBytes is activated if channel group record size (CGrecordLength) is bigger than the computed record size (line 501) Could you check what are the values with a print and compare with MDFvalidator (in channel group) ?

[rolfub]

Hello Aymeric,

here are the results from the debugging session:

self.CGrecordLength = 11 self.recordLength = 10

Length of record without record ID (Byte) = 11 <-- Vector MDF Validator

Rolf

[ratal]

Hi Rolfub, self.recordLength = 10 is what you get when calculating from Channel.posBit. But channel group says 11 bytes --> There is one hidden byte at the beginning of the record but Record.hiddenBytes is still false. I cannot understand as on line 501: if self.CGrecordLength > self.recordLength: self.hiddenBytes = True hiddenBytes should be activated ! Can you please find out why this is not activated ? I can't really go further without the file (I double checked and related code is limited, I could not see what is wrong) Aymeric

[rolfub]

Hello Aymeric,

sorry, it was my fault. I have made a typing error yesterday. :-( Here are the corrected values:

self.CGrecordLength = 11 self.recordLength = 12

Length of record without record ID (Byte) = 11 <-- Vector MDF Validator

Rolf

[ratal]

Can you try to put line 490 aligned with line 473 ?: elif len(self) ==1 and channel.posBitBeg >= 8: self.hiddenBytes = False

[rolfub]

Hello Aymeric,

pardon, where should I insert the line "elif len(self) ==1 and channel.posBitBeg >= 8: self.hiddenBytes = False"? Isn't "self.hiddenBytes" already "False" by default?

Rolf

[ratal]

Hi Rolfub,

in record.loadinfo

        if len(self) > 1:
            # all channels are already ordered in record based on byte_offset and bit_offset
            # so just comparing with previous channel
            if channel.byteOffset >= self[-2].byteOffset and \
                    channel.posBitBeg < 8 * (self[-2].byteOffset + self[-2].nBytes) and \
                    channel.posBitEnd > 8 * (self[-2].byteOffset + self[-2].nBytes):  # not byte aligned
                self.byte_aligned = False
            if channel.posBitBeg >= 8 * self[-2].byteOffset \
                    and channel.posBitEnd <= 8 * (self[-2].byteOffset + self[-2].nBytes):  # bit(s) in byte(s)
                embedded_bytes = True
                if self.recordToChannelMatching: # not first channel
                    self.recordToChannelMatching[channel.recAttributeName] = self.recordToChannelMatching[self[-2].recAttributeName]
                else: # first channels
                    self.recordToChannelMatching[channel.recAttributeName] = channel.recAttributeName
                    self.numpyDataRecordFormat.append(channel.RecordFormat)
                    self.dataRecordName.append(channel.recAttributeName)
                    self.recordLength += channel.nBytes
        elif len(self) == 1 and Channel.posBitBeg >= 8 : # first channel starts within first byte ?
            self.hiddenBytes = True

It should look like above

[rolfub]

Hello Aymeric,

there is a new exception after the change. Here is the python traceback: Traceback (most recent call last): File "C:\Users\XXX\Desktop\CANoe\CANoe_test.py", line 215, in file = mdfreader.mdf( "C:\Users\XXX\Desktop\CANoe\CANoe.mdf" ) File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdf.py", line 115, in init self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames) File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdfreader.py", line 360, in read self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False) File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdf3reader.py", line 934, in read3 buf.read(channelSet) # reads datablock potentially containing several channel groups File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdf3reader.py", line 750, in read data = self.loadUnSorted(nameList=channelSet) File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdf3reader.py", line 811, in loadUnSorted temp = self[recordID]['record'].readRecordBits(stream[position:position + self[recordID]['record'].CGrecordLength + 1], nameList) File "C:\Python27\lib\site-packages\mdfreader-0.2.5-py2.7-win32.egg\mdfreader\mdf3reader.py", line 663, in readRecordBits if 's' not in Channel.Format: AttributeError: Channel instance has no attribute 'Format'

Rolf

[ratal]

Hi Rolfub, Please check last commit Aymeric

[rolfub]

Hello Aymeric,

it works. :-) Thank you for your time and your patience!

At the Moment we use mdf3 but in the future we will use mdf4. Should we test mdf4 now or later?

Rolf

[ratal]

Hi Rolfub, You can try also mdf4. When issues are detected in one version, I generally check if this is also valid for the other version and I apply similar counter measure. But it should be more robust for mdf4 as I have since longer time unsorted example files. Regards Aymeric