Error reading MF4.1 file

[Salsapareille]

Hi,

sorry if I am not using the typical way of bug reporting but I am a very newbie on GitHub. I just tried the mdfreader on my MDF 4.1 files (measurement files from endurance tests) just to see if it was working and:

• I was capable to get an info object and use it, get a list of channels and so on
• by trying to read the data(mdfreader.mdf('file name'), I am getting an error message: IndexError: bitarray index out of range. The file is 1.4GB big.

My working environment is Anaconda 3.4.2.0 64bits on Win7.

Here the whole error message: f = mdfreader.mdf(r'C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4')

C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4 Unexpected error: (<class 'ImportError'>, ImportError("No module named 'mdfreader.dataRead'",), <traceback object at 0x0000000009696208>) dataRead crashed, back to python data reading

---

ImportError Traceback (most recent call last) C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in readBitarray(self, bita, channelList) 1178 try: # use rather cython compiled code for performance -> 1179 from .dataRead import dataRead 1180 for chan in range(len(self)):

ImportError: No module named 'mdfreader.dataRead'

During handling of the above exception, another exception occurred:

IndexError Traceback (most recent call last)

in () ----> 1 f = mdfreader.mdf(r'C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4') C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf.py in **init**(self, fileName, channelList, convertAfterRead, filterChannelNames) 104 self.fileName = fileName 105 if fileName is not None: --> 106 self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames) 107 108 C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdfreader.py in read(self, fileName, multiProc, channelList, convertAfterRead, filterChannelNames) 351 self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False) 352 else: # MDF version 4.x. Channel by channel reading implemented --> 353 self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False) 354 355 def write(self, fileName=None): C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in read4(self, fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames) 1364 channelList.extend(['ms', 'min', 'hour', 'day', 'month', 'year']) 1365 -> 1366 buf.read(channelList) # reads raw data from data block with DATA and DATABlock classes 1367 1368 # processing data from buf then transfer to self C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in read(self, channelList, zip) 367 recordID = list(self.keys())[0] 368 record = self[recordID]['record'] --> 369 self[recordID]['data'] = self.load(record, zip=None, nameList=channelList, sortedFlag=True) 370 for cn in record.VLSD: # VLSD channels 371 if channelList is None or record[cn].name in channelList: C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in load(self, record, zip, nameList, sortedFlag) 449 data_block['id'] = '##DT' # do not uncompress in DATABlock function 450 data_block['data'] = buf --> 451 temps['data'] = DATABlock(record, parent_block=data_block, channelList=nameList, sortedFlag=sortedFlag) 452 else: # empty datalist 453 temps['data'] = None C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in DATABlock(record, parent_block, channelList, sortedFlag) 94 return fromstring(parent_block['data'], dtype=record.numpyDataRecordFormat, shape=record.numberOfRecords, names=record.dataRecordName) 95 else: # record is not byte aligned or channelList not None ---> 96 return record.readBitarray(parent_block['data'], channelList) 97 else: # unsorted reading 98 print('not implemented yet unsorted data block reading', file=stderr) # to be implemented if needed, missing example file C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in readBitarray(self, bita, channelList) 1227 temp[i].extend(byte) 1228 else: # signed integer (two's complement), keep sign bit and extend with bytes -> 1229 temp = signedInt(temp, byte) 1230 nTrailBits = self[chan].nBytes*8 - self[chan].bitCount 1231 if self[chan].signalDataType in (2, 3) and \ C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in signedInt(temp, extension) 1203 extensionInv.setall(True) 1204 for i in range(self.numberOfRecords): # extend data of bytes to match numpy requirement -> 1205 signBit = temp[i][-1] 1206 if not signBit: # positive value, extend with 0 1207 temp[i].extend(extension) IndexError: bitarray index out of range If you are interested, I may try to give you the file leading to this failure - if it is possible to upload such big file on GitHub... Thanks!

[ratal]

Hi, It seems first that mdfreader was not installed properlly. During installation, it should compile dataRead cython file for your platform (first error). However, there is a fall back using bitarray module which is very slow but should be working-> the rest of the errors. I propose you the following:

1. try to compile the cython file dataRead (depending how you installed mdfreader: reinstall using pip or go in the package folder and do python setup.py install to trigger compilation)
2. send me the file that I can download it (google drive, dropbox, etc. ?) and I troubleshoot or at least send a .xml file of freely available mdfvalidator (vector site) to understand the file structure, maybe some specific architecture was not programmed yet. Aymeric

[Salsapareille]

Hi, cython was the keyword: I am now able to load and plot the MDF-signals. I tried the following:

1. re-install mdfreader using pip => was not working. Several packages named "mdfreader", "mdfconverter" were installed in the site-packages but without the files "dataRead.py" and the pyd-file.
2. Running python setup.py install deliver a better result from the point of view of package installation, but also I was getting the same failure. At least there is only one package called "mdfreader-0.2.2-py3.5-win-amd64.egg" installed inside site-packages. Also readData.py and dataRead.cp35-win_amd64.pyd are present in the root of the package folder.
3. Copying the files "readData.py" and "dataRead.cp35-win_amd64.pyd" into the folder "mdfreader" is helping. Of course I am not sure this is a good idea. Anyway until now the mdfreader seems to be able to read my mdf4-files. I can continue my tests now. Thank you very much for you help!

[ratal]

Glad I could help. However I do not understand why it did install properly out of the box. Anyway closed.