search

RecordEvolution / IMCtermite

Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS/imcSTUDIO and facilitates its storage in open source file formats
MIT License
27 stars 9 forks source link

Support for multi component datafields (XY-dataset) #30

Open jgoedeke opened 1 week ago

jgoedeke commented 1 week ago

Hi, I tried to parse a file with a single channel containing a XY-dataset. This means it contains a component group with two components one for the xdata (here a timestamp in seconds) and one for the ydata (here without unit).

It looks like the library does not parse multiple components properly, meaning it can only open timestamped ASCI and normal datasets. When having multiple components in one group the xdata is still filled via the xstepwidth from the CD key which is not applicable in this case.

https://github.com/RecordEvolution/IMCtermite/blob/befda072e5ffe038a681874bcb047e8f1efd3f59/lib/imc_channel.hpp#L478-L482

Steps to reproduce:

>>> from IMCtermite import imctermite
>>> imc = imctermite('XY_dataset_example.dat'.encode())
>>> channel = imc.get_channels(True)[0]
>>> channel['xdata'][-5:]
[13089, 13090, 13091, 13092, 13093]
>>> channel['ydata'][-5:]
[3169029, 2796202, 2796202, 2796202, 2982616]

XY_dataset_example.zip

I am using version 2.0.20 with python 3.11 on linux.

I am willing to create a PR but I am wondering how we can determine which blocks belong together. Because for parsing the xdata in the same way as the ydata for XY-datasets we have to make sure we have to corresponding CC, CP, CD, NT, CR, CN and Cb keys. Does this require a change in channel parsing? Maybe by adding component group and component vectors to represent the right structure?

Best regards, Jan

mario-fink commented 1 week ago

Hi Jan,

thanks for using the package and creating the issue!

You're welcome to provide a PR to support XY data. To do this you have to collect all CC keys/blocks associated to the preceding CG block. For instance, in your example XY_dataset_example.dat the |CG,1,5,2,2,2; block features 2 CC blocks (first 2) and the field type 2 (second 2) corresponding to XY data with multiple y-datasets with a common x-timedataset. Listing all blocks associated to a group is done here and ff.

https://github.com/RecordEvolution/IMCtermite/blob/befda072e5ffe038a681874bcb047e8f1efd3f59/lib/imc_raw.hpp#L206-L220

Best regards, Mario