search

pyomeca / ezc3d

Easy to use C3D reader/writer for C++, Python and Matlab
https://pyomeca.github.io/Documentation/ezc3d/index.html
MIT License
142 stars 44 forks source link

Combined C3D files exhibiting bug with POINT data and ANALOG data merged together #294

Closed jagusto33 closed 9 months ago

jagusto33 commented 1 year ago

Hello,

I have created new C3D files in MATLAB (Windows 10 computer) from existing marker data (POINTS) in existing .c3d files and creating a new .c3d file by adding in EMG data (ANALOG). I have no errors when creating the new .c3d file with my merged data, but when I open some of these .c3d files in a processer (Visual 3D), both my marker data and analog data seem to appear and disappear throughout the recording. For example, during a quiet standing task collect for 30 seconds, my markers will sequentially disappear and reappear on the research subject (artificial).

I have attached an example .c3d file of this quiet standing task with ONLY the Point data (displaying correctly) and the same exact trial merged with EMG data to show the issues. Any idea on what might be happening when I am using the ezc3d read and write commands that could somehow be affecting the data?

P.S. I could not attach the .c3d file type here in GitHub, DM me to share the example files: jonathan_a_gustafson@rush.edu

jagusto33 commented 11 months ago

Hello, just checking in to see if there was any guidance on this issue we were facing with creating a new .c3d file by merging both point and analog data?

pariterre commented 11 months ago

My appologies, I completely forgot that issue! I am looking at this at the moment, let you know quickly!

pariterre commented 11 months ago

I again! After having a look at the files you sent me, I'd say it is not just a matter of the data to skip, but they are all actually completely messed up...

My first guess would be that there is an incompatibility in the number of data and the frame rate. Is this intended that the frame rate for the markers to be 300.003?

If you can provide me with the code that generated the merged data, it could help (no need to send actual data for the analog, I can use random to generate some, as long as I know to expected size!

jagusto33 commented 11 months ago

Hello,

Attached are two scripts (and corresponding functions) I wrote to navigate through my existing data. Some notes for context: -Each script operates on either .mat files of Analog data that need to be combined with existing .c3d data or create a new c3d file of the EMG data alone -Our repository of EMG analog data and Mocap data were in .mat and .c3d file formats, respectively; my goal was to import the .mat Analog data and the existing .c3d mocap data for the same trial and "merge" the datasets together in a new c3d -*Of note: you commented on the frequency for some trials. I spoke with our research engineer and he mentioned that when this data was collected, for some reason the system was not "restricted" to whole integer values for sampling rate, which is why some trials the frame rate is as you mention. In my scripts, I thought I could generate the correct ratio by "rounding" the mocap sample rate and padding the data required on the EMG (padding at the end) to meet the requirements for the Analog/Video ratio variable...this might be a source of issue, in that I am rounding our mocap frame rate instead of calculating the data points off the original frame rate. Any thoughts on how to handle this?

Let me know if you have any questions and I'd be happy to hop on a Zoom call if it would be helpful!

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Pariterre @.> Sent: Wednesday, September 27, 2023 12:46 PM To: pyomeca/ezc3d @.> Cc: Jonathan Gustafson @.>; Author @.> Subject: Re: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Rush Email Security

This email originated from outside of RUSH. Do not click links or attachments unless you recognize the sender and know that the content is safe. RUSH will never ask for user ID information via email.

I again! After having a look at the files you sent me, I'd say it is not just a matter of the data to skip, but they are all actually completely messed up...

My first guess would be that there is an incompatibility in the number of data and the frame rate. Is this intended that the frame rate for the markers to be 300.003?

If you can provide me with the code that generated the merged data, it could help (no need to send actual data for the analog, I can use random to generate some, as long as I know to expected size!

- Reply to this email directly, view it on GitHubhttps://github.com/pyomeca/ezc3d/issues/294#issuecomment-1737829466, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX6KBS2BXHGJIGN7EYRTIDLX4RQ4ZANCNFSM6AAAAAA3AIVCMU. You are receiving this because you authored the thread.Message ID: @.**@.>>

jagusto33 commented 11 months ago

Hello,

Just wanted to check back in and make sure you received my scripts and if it would be worth trying to discuss this and any insight into why the .c3d files were coming out improperly when I was trying to combine my analog (EMG) and point data.

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Jonathan Gustafson Sent: Thursday, September 28, 2023 5:28 PM To: pyomeca/ezc3d @.>; pyomeca/ezc3d @.> Cc: Author @.***> Subject: RE: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Hello,

Attached are two scripts (and corresponding functions) I wrote to navigate through my existing data. Some notes for context: -Each script operates on either .mat files of Analog data that need to be combined with existing .c3d data or create a new c3d file of the EMG data alone -Our repository of EMG analog data and Mocap data were in .mat and .c3d file formats, respectively; my goal was to import the .mat Analog data and the existing .c3d mocap data for the same trial and "merge" the datasets together in a new c3d -*Of note: you commented on the frequency for some trials. I spoke with our research engineer and he mentioned that when this data was collected, for some reason the system was not "restricted" to whole integer values for sampling rate, which is why some trials the frame rate is as you mention. In my scripts, I thought I could generate the correct ratio by "rounding" the mocap sample rate and padding the data required on the EMG (padding at the end) to meet the requirements for the Analog/Video ratio variable...this might be a source of issue, in that I am rounding our mocap frame rate instead of calculating the data points off the original frame rate. Any thoughts on how to handle this?

Let me know if you have any questions and I'd be happy to hop on a Zoom call if it would be helpful!

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Pariterre @.**@.>> Sent: Wednesday, September 27, 2023 12:46 PM To: pyomeca/ezc3d @.**@.>> Cc: Jonathan Gustafson @.**@.>>; Author @.**@.>> Subject: Re: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Rush Email Security

This email originated from outside of RUSH. Do not click links or attachments unless you recognize the sender and know that the content is safe. RUSH will never ask for user ID information via email.

I again! After having a look at the files you sent me, I'd say it is not just a matter of the data to skip, but they are all actually completely messed up...

My first guess would be that there is an incompatibility in the number of data and the frame rate. Is this intended that the frame rate for the markers to be 300.003?

If you can provide me with the code that generated the merged data, it could help (no need to send actual data for the analog, I can use random to generate some, as long as I know to expected size!

- Reply to this email directly, view it on GitHubhttps://github.com/pyomeca/ezc3d/issues/294#issuecomment-1737829466, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX6KBS2BXHGJIGN7EYRTIDLX4RQ4ZANCNFSM6AAAAAA3AIVCMU. You are receiving this because you authored the thread.Message ID: @.**@.>>

pariterre commented 11 months ago

Hi again!

Sorry! I just sent you an email saying that I had everything, but it was not the case (I mixed up two issues that I was working on). I indeed did NOT receive the script. I was somehow block, don't know if it was from my side or yours... Could you send it again!

Sorry for the inconvenience!

pariterre commented 11 months ago

Any thoughts on how to handle this?

I think this could be done by manually changing the value of 300.003 in the POINT:RATE parameters and making sure ANALOG:RATE matches the ratio would work. But if I understand well, this is precisely what you did? Apart from that, I am not sure how to handle this. If it does not work, please send me again your script :)

pariterre commented 11 months ago

Here is a completely simulated test based on what was not working from your code. I do not have access today to Nexus, so if you could try to load each of the produced c3d. I suspect the only one which will behave as expected is the when both points and analogs rate are rounded.

clc;clear;

pointRate = 300.003;
nbPoints = 40;
nbFrames = 100;

analogFrameRatio = 10;
nbAnalogs = 8;

conditions = {
    [false, false], [false, true], [true, false], [true, true]
};

for i = 1:length(conditions)
    % Prepare the test conditions
    roundPointRate = conditions{i}(1);
    roundAnalogRate = conditions{i}(2);

    if roundPointRate
        conditionPointRate = round(pointRate);
    else
        conditionPointRate = pointRate;
    end

    if roundAnalogRate
        conditionAnalogRate = round(pointRate) * analogFrameRatio;
    else
        conditionAnalogRate = pointRate * analogFrameRatio;
    end

    % Create a c3d from scratch
    mocapData = ezc3dRead();

    pointNames = cell(nbPoints, 1);
    for j = 1:nbPoints
        pointNames{j} = sprintf('Point_%d', j);
    end

    analogNames = cell(nbAnalogs, 1);
    for j = 1:nbAnalogs
        analogNames{j} = sprintf('Analog_%d', j);
    end

    % Adjust the parameters
    mocapData.parameters.POINT.LABELS.DATA = pointNames;
    mocapData.parameters.POINT.RATE.DATA = conditionPointRate;

    mocapData.parameters.ANALOG.LABELS.DATA = analogNames;
    mocapData.parameters.ANALOG.RATE.DATA = conditionAnalogRate;

    % Fill the data structure
    mocapData.data.points = rand(3, nbPoints, nbFrames);
    mocapData.data.analogs = rand(nbFrames * analogFrameRatio, nbAnalogs);

    % Write the c3d
    ezc3dWrite(sprintf('exported_roundPoint%d_roundAnalog%d.c3d', roundPointRate, roundAnalogRate), mocapData);

end
jagusto33 commented 6 months ago

Hello,

I know you closed this back in December; I had been quite buys with some other projects and am circling back to this!

I ran your script below and as expected, the only C3D file that worked correctly was the one where the rate data were rounded. Does this mean to resolve some of my issues, I should explicitly change the point.rate parameters data in each of the existing files to make sure they are rounded? I can see why my original script would have issue, because I was dealing with a rounded analog sampling rate (usually either 1500 Hz or 3000 Hz) and the point rate data would sometimes be 150.0015 Hz in the original C3D that I was trying to open and merge analog data with.

I just want to be sure that I am not possibly corrupting data by manually changing the point rate data.

Thank you for any insight you can provide!

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Pariterre @.> Sent: Tuesday, October 10, 2023 3:01 PM To: pyomeca/ezc3d @.> Cc: Jonathan Gustafson @.>; Author @.> Subject: Re: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Rush Email Security

This email originated from outside of RUSH. Do not click links or attachments unless you recognize the sender and know that the content is safe. RUSH will never ask for user ID information via email.

Here is a completely simulated test based on what was not working from your code. I do not have access today to Nexus, so if you could try to load each of the produced c3d. I suspect the only one which will behave as expected is the when both points and analogs rate are rounded.

clc;clear;

pointRate = 300.003;

nbPoints = 40;

nbFrames = 100;

analogFrameRatio = 10;

nbAnalogs = 8;

conditions = {

[false, false], [false, true], [true, false], [true, true]

};

for i = 1:length(conditions)

% Prepare the test conditions

roundPointRate = conditions{i}(1);

roundAnalogRate = conditions{i}(2);

if roundPointRate

    conditionPointRate = round(pointRate);

else

    conditionPointRate = pointRate;

end

if roundAnalogRate

    conditionAnalogRate = round(pointRate) * analogFrameRatio;

else

    conditionAnalogRate = pointRate * analogFrameRatio;

end

% Create a c3d from scratch

mocapData = ezc3dRead();

pointNames = cell(nbPoints, 1);

for j = 1:nbPoints

    pointNames{j} = sprintf('Point_%d', j);

end

analogNames = cell(nbAnalogs, 1);

for j = 1:nbAnalogs

    analogNames{j} = sprintf('Analog_%d', j);

end

% Adjust the parameters

mocapData.parameters.POINT.LABELS.DATA = pointNames;

mocapData.parameters.POINT.RATE.DATA = conditionPointRate;

mocapData.parameters.ANALOG.LABELS.DATA = analogNames;

mocapData.parameters.ANALOG.RATE.DATA = conditionAnalogRate;

% Fill the data structure

mocapData.data.points = rand(3, nbPoints, nbFrames);

mocapData.data.analogs = rand(nbFrames * analogFrameRatio, nbAnalogs);

% Write the c3d

ezc3dWrite(sprintf('exported_roundPoint%d_roundAnalog%d.c3d', roundPointRate, roundAnalogRate), mocapData);

end

- Reply to this email directly, view it on GitHubhttps://github.com/pyomeca/ezc3d/issues/294#issuecomment-1756138392, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX6KBSYLV5XC45LUJMJTNLLX6WSQXAVCNFSM6AAAAAA3AIVCMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJWGEZTQMZZGI. You are receiving this because you authored the thread.Message ID: @.**@.>>

pariterre commented 6 months ago

Hi there! In any case, if you plan to modify a C3D, I'd make sure to back it back somehow. Since the RATE is expected to be a whole number, this is indeed the proper solution, from the standpoint of ezc3d at least. Because of the internal structure of ezc3d, there is no way ezc3d can handle non integer ratio rate between points and analogs.

I have no clue though if this discrepancy will introduce error in your data or not...

jagusto33 commented 6 months ago

Here is an example of what happened when I tried to merge Analog data (captured at 3000 Hz) with an existing C3D format Point data (captured at 150.001 Hz) WITHOUT rounding the point rate data to 150Hz.

My script was trying to make sure the total number of frames in the analog data for this trial (15507 frames) rounded to a integer value with the video-based frames for the same trial (804 frames). Obviously, dividing the analog_frame / video_frame = 19.2873, a non-integer value that will cause an error when trying to build the .c3d.

What I have done is either "padded" my analog data with "0" values if I needed more to get close to up-rounding to the nearest integer OR deleted analog data from the end of my data range to round down to the nearest integer.

In the example above, I removed the last 231 datapoints from the Analog signal to get my new_analog_frame / video_frame = 19, thus being a whole number. The C3D would then be created without issue because I had a whole integer value for my RATIO.

However, I never adjusted / edited the frame rate data to be a round sampling rate (e.g. 150 Hz), and so when I would load the combined .c3d, you would see issues like in the video.

I'm troubleshooting this now to see if just changing the POINT.RATE parameter fixes this, or if this is an underlying issue with how I am editing my ANALOG data frames to comply with the RATIO requirements.

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Pariterre @.> Sent: Tuesday, February 27, 2024 10:45 AM To: pyomeca/ezc3d @.> Cc: Jonathan Gustafson @.>; Author @.> Subject: Re: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Rush Email Security

This email originated from outside of RUSH. Do not click links or attachments unless you recognize the sender and know that the content is safe. RUSH will never ask for user ID information via email.

Hi there! In any case, if you plan to modify a C3D, I'd make sure to back it back somehow. Since the RATE is expected to be a whole number, this is indeed the proper solution, from the standpoint of ezc3d at least. Because of the internal structure of ezc3d, there is no way ezc3d can handle non integer ratio rate between points and analogs.

I have no clue though if this discrepancy will introduce error in your data or not...

- Reply to this email directly, view it on GitHubhttps://github.com/pyomeca/ezc3d/issues/294#issuecomment-1967071098, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX6KBS26QOTCSBDFNM7Z74TYVYESPAVCNFSM6AAAAAA3AIVCMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXGA3TCMBZHA. You are receiving this because you authored the thread.Message ID: @.**@.>>

jagusto33 commented 6 months ago

Ok, so I figured out the bug! It was my script attempting to round to a multiplier based off the frames and not the original SAMPLING RATE for the video and analog data. So in my example below, where the AV RATIO should be 20 (3000 Hz / 150 Hz), my script was rounding to the nearest whole integer from the total collect frames, which was 19. This inconsistency of having only enough analog to video data frames at that ratio was causing the odd bug seen in my attached file.

I will re-run my entire routine on my database and hopefully should have resolved the issue 😊

Thank you for all your help on this! I have a much deeper appreciation for the .C3D file format after this exercise!

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Jonathan Gustafson Sent: Tuesday, February 27, 2024 11:14 AM To: pyomeca/ezc3d @.>; pyomeca/ezc3d @.> Cc: Author @.***> Subject: RE: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Here is an example of what happened when I tried to merge Analog data (captured at 3000 Hz) with an existing C3D format Point data (captured at 150.001 Hz) WITHOUT rounding the point rate data to 150Hz.

My script was trying to make sure the total number of frames in the analog data for this trial (15507 frames) rounded to a integer value with the video-based frames for the same trial (804 frames). Obviously, dividing the analog_frame / video_frame = 19.2873, a non-integer value that will cause an error when trying to build the .c3d.

What I have done is either “padded” my analog data with “0” values if I needed more to get close to up-rounding to the nearest integer OR deleted analog data from the end of my data range to round down to the nearest integer.

In the example above, I removed the last 231 datapoints from the Analog signal to get my new_analog_frame / video_frame = 19, thus being a whole number. The C3D would then be created without issue because I had a whole integer value for my RATIO.

However, I never adjusted / edited the frame rate data to be a round sampling rate (e.g. 150 Hz), and so when I would load the combined .c3d, you would see issues like in the video.

I’m troubleshooting this now to see if just changing the POINT.RATE parameter fixes this, or if this is an underlying issue with how I am editing my ANALOG data frames to comply with the RATIO requirements.

Jonathan

Jonathan A. Gustafson, PhD Assistant Professor | IBTS Fellowship Research Director Orthopedic Surgery Rush University Medical Center Phone: 708-522-2227

From: Pariterre @.**@.>> Sent: Tuesday, February 27, 2024 10:45 AM To: pyomeca/ezc3d @.**@.>> Cc: Jonathan Gustafson @.**@.>>; Author @.**@.>> Subject: Re: [pyomeca/ezc3d] Combined C3D files exhibiting bug with POINT data and ANALOG data merged together (Issue #294)

Rush Email Security

This email originated from outside of RUSH. Do not click links or attachments unless you recognize the sender and know that the content is safe. RUSH will never ask for user ID information via email.

Hi there! In any case, if you plan to modify a C3D, I'd make sure to back it back somehow. Since the RATE is expected to be a whole number, this is indeed the proper solution, from the standpoint of ezc3d at least. Because of the internal structure of ezc3d, there is no way ezc3d can handle non integer ratio rate between points and analogs.

I have no clue though if this discrepancy will introduce error in your data or not...

— Reply to this email directly, view it on GitHubhttps://github.com/pyomeca/ezc3d/issues/294#issuecomment-1967071098, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX6KBS26QOTCSBDFNM7Z74TYVYESPAVCNFSM6AAAAAA3AIVCMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXGA3TCMBZHA. You are receiving this because you authored the thread.Message ID: @.**@.>>