search

danielhrisca / asammdf

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files
GNU Lesser General Public License v3.0
648 stars 226 forks source link

Filtering of the group and channel changes the structure of the output mf4 file #334

Closed ankesh90 closed 4 years ago

ankesh90 commented 4 years ago

Task is to extract the flexray channel from the input mf4 file. Version 4.10

For this I have used the asammdf 5.15.1 GUI feature where after importing the mf4 file and selecting the particular group related to flex ray channel I am able to filter and save it as mf4 file. After comparing the input and output mf4 file I can see there are lot of changes in the structure as a result my further analysis gets failed as it is expecting the structure to be same as input file. Could you please let me know why the structure is coming different and what steps can be taken to have the output structure same as the input structure.

I have attached the screenshot of the input and output structure
input_structure output_structure

danielhrisca commented 4 years ago

Hello Ankesh,

can you send your file?

danielhrisca commented 4 years ago

@ankesh90 the development branch code should work a it better at preserving the channel structure

ankesh90 commented 4 years ago

@danielhrisca Currently I am in conversation with the data owner and exploring the possibility if file can be shared or not. Please give me sometime. Regarding the development branch, I will clone it and see if it preserves the structure. I will share the result soon Thank you

ankesh90 commented 4 years ago

@danielhrisca I cannot use the latest development branch it is throwing the bustype error while creating the MDF object. I was able to extract the flexray data using filter function of the MDF module (Version 5.19 master)

When I analysed the data in the Canoe vector tool I can see that not all the channels are extracted for the flexray frame.

I have used the command MDF.filter([(None, 5 , 1)]) where 5 is the group index and 1 is the channel index but I am not sure if this is the correct command and also if channel index = 1 is correct or not. Group index and channel index value you can reference from the previous screenshots.

Attaching the screenshot of the input raw file analysed in the canoe tool Flexray stats shows that there are 5000 frames per second and total frames are 8425 stat_trace_filtered_data

Attaching the screenshot of the filtered fille analysed in the canoe tool Flexray stats shows that there are only 37 frames per second and total frames are 63 and also trace window shows less entry of the channels stat_trace_raw_data

danielhrisca commented 4 years ago

It would help if you shared the full error traceback message

ankesh90 commented 4 years ago

It would help if you shared the full error traceback message

Yes that I will paste here but meanwhile could you please tell if I am extracting the flexray correctly using filter function from asammdf 5.19 version from the master

danielhrisca commented 4 years ago

MDFValidator uses 1-based indexing for data groups, and this library uses 0-based indexing. SO you should actually call MDF.filter([(None, 4 , 1)])

ankesh90 commented 4 years ago

It would help if you shared the full error traceback message

Here it is error

ankesh90 commented 4 years ago

MDFValidator uses 1-based indexing for data groups, and this library uses 0-based indexing. SO you should actually call MDF.filter([(None, 4 , 1)])

I have used index 4 only sorry for the confusion.

danielhrisca commented 4 years ago

Here it is

The new dev code should have this fixed now

ankesh90 commented 4 years ago

Hi Daniel, Thanks for fixing the issue in development branch. The structure of the extracted Flex ray file using dev branch is same as compared to the master branch. The output file has missing signals when analysed using vector canoe tool. The screenshot is attached in the previous comment.

This is the feedback I got from the vector team

the file does not contain a valid BusLog event structure => the signals are contained in “flat” format instead of using a structure. In addition the bus channel information is missing (this is required). Recommended: the bus channel value should be > 0.

As the file cannot be shared due to the company policy. Can we have the live session for the discussion on this issue ?. It is one of the highest priority issue, I will arrange for the meeting let me know the feasible time.

danielhrisca commented 4 years ago

Hello @ankesh90

I've fixed the problem with flattening the structures, please give it a try with the development branch code

ankesh90 commented 4 years ago

Hello Daniel, Thanks for fixing the above issue. Still I am not able to load the missing signal. I am attaching the screenshot. Could you please explain why the marked structure is coming different in the input and output file

Input file input_file

output file filtered_file

danielhrisca commented 4 years ago

It's just the way the raw data is stored in the file

ankesh90 commented 4 years ago

It's just the way the raw data is stored in the file

Yes that's the point why the input file Data block and FLX_Frame.Databytes structure is different from the filtered file. As filter operation is done on the input file. Why the structure is not coming same ? What can be done to keep it same? Also if you observe the DataBlock length it is coming different in both the Data Group. In the input file it is 210312 and in output file it is 1910140.

danielhrisca commented 4 years ago

In input file the actual Ethernet payload is stored as a signal data block referenced by the FLX_Frame.DataBytes channel. After filtering the payload is stored in the DataGroup blocks directly, hence the difference

ankesh90 commented 4 years ago

In input file the actual Ethernet payload is stored as a signal data block referenced by the FLX_Frame.DataBytes channel. After filtering the payload is stored in the DataGroup blocks directly, hence the difference

Thanks for the info. As communicated by the Vector team that Input file has the correct structure required for the Canoe tool Can we tweak the filter function such that structure is same or if there is any other process which can be followed is also fine

danielhrisca commented 4 years ago

Can you try again with the new development code. It should preserve the original source information

ankesh90 commented 4 years ago

Can you try again with the new development code. It should preserve the original source information

I can see there are 2 development branches, origin/development and origin/4.x.y-development Till now I have used origin/development is this correct ?

danielhrisca commented 4 years ago

Can you try again with the new development code. It should preserve the original source information

I can see there are 2 development branches, origin/development and origin/4.x.y-development Till now I have used origin/development is this correct ?

Yes

ankesh90 commented 4 years ago

Hi Daniel, Sorry didn't reply earlier. The develop changes didn't work. I can see without the input file it is difficult for you to change the filter function.. I am able to get the input file which I can share with you.

This file I can open in Canoe tool. Please focus on DataGroup5 (FlexRay)

  1. if you can change the filter function in a way that structure of the Data Block and FLX_Frame.Databytes of the input and filtered file should be same.
  2. The Block length of the Data block and FLX_Frame.Databytes in filtered file is also same In this case 644448 and 1717218 respectively. Let me know if you need any other details.
    Thank you 20200416T112109Z_223-1164__468_Full_Recording_NoCam_Filtered.zip
danielhrisca commented 4 years ago
  1. if you can change the filter function in a way that structure of the Data Block and FLX_Frame.Databytes of the input and filtered file should be same.

This is not mandatory according to the standard. The way the lib saved the DataBytes directly in the channel group record is perfectly fine.

The probably simplest way is to model the "DataBytes" channel as ordinary value channel (cn_type = 0) with data type BYTE_ARRAY (cn_data_type = 10) and a size equal to the maximum number of bytes possible for this frame type (e.g. cn_bit_count = 8 x number of bytes)

ankesh90 commented 4 years ago
  1. if you can change the filter function in a way that structure of the Data Block and FLX_Frame.Databytes of the input and filtered file should be same.

This is not mandatory according to the standard. The way the lib saved the DataBytes directly in the channel group record is perfectly fine.

The probably simplest way is to model the "DataBytes" channel as ordinary value channel (cn_type = 0) with data type BYTE_ARRAY (cn_data_type = 10) and a size equal to the maximum number of bytes possible for this frame type (e.g. cn_bit_count = 8 x number of bytes)

okay even if we keep the strucure as it is ...Total length of the Datablock should be (644448 + 1717218) but I can see that after filtering the Datablock length is different.. could you please confirm that we don't loose any data ?

danielhrisca commented 4 years ago

Extracting the last DataBytes sample using asammdf form the original and filtered files

from asammdf import MDF

mdf = MDF(r"d:\TMP\20200416T112109Z_223-1164__468_Full_Recording_NoCam_Filtered.mf4")

ch = mdf.get("FLX_Frame")
print(bytes(ch.samples["FLX_Frame.DataBytes"][-1][:ch.samples["FLX_Frame.DataLength"][-1]]).hex())

mdf = MDF(r"d:\TMP\filt_flx_2.mf4")

ch = mdf.get("FLX_Frame")
print(bytes(ch.samples["FLX_Frame.DataBytes"][-1][:ch.samples["FLX_Frame.DataLength"][-1]]).hex())
25b60a246e4c54528700000010fe7f0708000150e9f854abe16f0cf8fdfdef07fd7b1b00f13304ca
25b60a246e4c54528700000010fe7f0708000150e9f854abe16f0cf8fdfdef07fd7b1b00f13304ca

And the last DataBytes sample in CANape image