search

ratal / mdfreader

Read Measurement Data Format (MDF) versions 3.x and 4.x file formats in python
Other
169 stars 73 forks source link

Unable to merge two files, possibly due to hardcoded 'master' master channel name #184

Closed darth3PO closed 3 years ago

darth3PO commented 4 years ago

Python version

3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]

Platform information

Windows-10-10.0.16299-SP0

Numpy version

1.18.1

mdfreader version

4.0

Description

Unable to merge two mdf files. Files have no common channels, but were recorded simultaneously to different computers. One is version 300 and another 310.

yop = mdfreader.Mdf(file_name=r'File1.dat')
poy = mdfreader.Mdf(file_name=r'File2.dat')

yop.resample(sampling=0.5)
poy.resample(sampling=0.5)

After resampling, yop and poy each have one master channel, but they are named from one of the channels in the file, not 'time' or 'master'.

yop.masterChannelList.keys() ['time_1']

poy.masterChannelList.keys() ['TEST_TIME_1']

I've tried merging in each direction: combined = yop.merge_mdf(poy) and combined = poy.merge_mdf(yop) and both will yield the same error:

File "C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdfreader.py", line 1293, in merge_mdf: refill = empty(len(mdf_class.get_channel_data('master')))

TypeError: object of type 'NoneType' has no len()

Neither mdf has a master channel named master.

Is 'master' a reserved name? or should resample() force a name rather than inherit a name from the file?

ratal commented 4 years ago

Thanks for pointing out. Yes, the code is rather old, I will revise it and remove reference to 'master' channel name. However, I do not think merge_mdf() is what you are thinking. It is concatenating (sorry, name is misleading, I should change it), meaning it is putting 2 files/mdf one after the other in time. Basically biggest use case is to have one logger that records files once big enough for a long recording period of time, for endurance runs for instance. Having 2 parallel loggers is kinda new use case -> new method should be written. But if you want a quick fix, you could use pandas conversion and use the merge method. Otherwise, just doing a quick yop.update(poy) could do it for your use case but you would have inconsistencies in attribute .masterChannelList to fix

ratal commented 4 years ago

In dev branch you will find merge_mdf() has changed name into concat_mdf() and a new merge_mdf() appears that answers your use case -> you can try it.

ecalpy commented 3 years ago

Hello Mr. Rateau,

I'm having a similar issue, but instead with data from the same vehicle and have all common channels.

Error text: _File "C:\Users\ecalder.conda\envs\ecalpy\lib\site-packages\mdfreader\mdf4reader.py", line 1556, in _get_channel_data4 vector = self.get_channel(channel_name)[dataField]

KeyError: 'data'_

Will your updated concat_mdf()/merge_mdf() code help me as well?

Thanks, Ecal

ratal commented 3 years ago

Hi Ecal, Did you try with last version 4.1 of mdfreader ? It includes concat_mdf() that should fit your need. In my tests, I tried to concatenate twice the same file and no issue. If your issue still exists, can you please detail what you are doing ?

ratal commented 3 years ago

I guess issue was solved after 5 months, do hesitate to reopen if not.