Large NaN values from the 3D keypoints trc file

[MeeroDLSU]

Good day!

My fellow students are using pose2sim as one of the main pipelines for our thesis. We love all of the work you've put into this project and appreciate all the useful features in pose2sim. We successfully adopted our own OpenPose 2d Keypoints data and got a working .trc file for OpenSim usage. However, upon further inspection of the .trc file, we noticed that there are many sequential NaN values in the trc file we acquired. Here are the details of what we did so far.

How we got our data: 1) Our data contains a participant walking from Point A to Point B and vice versa, all in one video recording. 2) We've manually synchronized our video recordings from the three cameras (all in different perspectives). 3) Upon performing camera calibration before data gathering, we get an RMS of the ff

What the generated trc file looks like:

As in the picture, many "burst" nan values exist in the trc file. We manually cut the video frames with these nan values in the 3D coordinates as a temporary workaround to run the trc file into OpenSim.

What we've tried so far:

Under the assumption of the NaN values coming from OpenPose problematic keypoints, we made a Python interpolating script that will interpolate missing values from the OpenPose 2D keypoints (values that are 0,0,0 for x,y,c). We were able to fill the values and recontruct new json files to feed into pose2sim. However, upon getting the new trc file, these burst NaN values are still present.

As mentioned, trimming from a certain point to drop these values is our temporary solution. We've noticed that these NaN frames correspond to the part of the video where the participant turns around to walk back to a certain point. This may be attributed to a spike of noise when OpenPose fails to correctly assign keypoints as the participant is turning around.

Our Questions: 1) What are the different factors that cause the triangulation.py to give a NaN value for a certain row & column? 2) Based on this post, what are the things that we can implement/fix in order to have consistent results?

We would appreciate any help in pointing us in the right direction!

[davidpagnon]

Hi, and thank you for your appreciation!

Large NaN values come when Pose2Sim does not manage to triangulate points.

• it is often due to bad calibration, but yours is not bad so I don't expect it to be the cause
• it can also be caused by the person being seen by less than 2 cameras. If this is the case, you will have to put the cameras further away, to use wider lenses, or to make the subject turn back earlier
• you can start by skipping the person association step, and go straight to triangulation. Then, did you try to visualize your results in OpenSim or Blender? What does it look like?
• Can you also open your trc file in Excel, for example, to check if some keypoints are systematically missed?
• Which pose estimation model did you use? Is it the same one as the one you specified in the Config.toml file?
• You can also set interp_if_gap_smaller_than to 1000 or more, so that it interpolates even if there is a large gap.
• Also, try to increase the value of reproj_error_threshold_triangulation and to decrease the one for likelihood_threshold_triangulation.

P.S.:

• We've manually synchronized our video recordings from the three cameras (all in different perspectives).

I just released yesterday a new version to automatically synchronize cameras from the results of pose estimation. If you want to test it and tell me how it goes I'd be interested! To install the new version, do pip install pose2sim -U

[MeeroDLSU]

Hello! We really appreciate the time you're giving for helping us. We will try the following points that you've suggested, especially the tuning "reproj_error_threshold_triangulation" and "likelihood_threshold_triangulation" values.

- it is often due to bad calibration, but yours is not bad so I don't expect it to be the cause

Yes, we've actually been able to render in OpenSim the skeleton model of a trimmed version of the faulty trc file.

- it can also be caused by the person being seen by less than 2 cameras. If this is the case, you will have to put the cameras further away, to use wider lenses, or to make the subject turn back earlier

This is likely not the case since in all three recorded videos, the participant's body fully captured by the cameras at any point.

- you can start by skipping the person association step, and go straight to triangulation. Then, did you try to visualize your results in OpenSim or Blender? What does it look like?

When we actually try to use the trc file with NaN values, OpenSim gives us an error and we aren't able to start the rendering process at all. Here is the error given by OpenSim.

InverseKinematicsTool Failed: Error reading rows in file '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/Pose2Sim/Demo/S01_Test_Session/S01_P01_Dom/S01_P01_T00/pose-3d/S01_P01_T00_0-260.trc'. Unexpected number of columns in line 42. Expected = 65. Received = 64. 
    Thrown at TRCFileAdapter.cpp:185 in extendRead().

If we trim and omit frames that have NaN values, we were actually successful in rendering the Skeleton Model. The following video below is what we've got by part selecting frames 2-40. (This was without Marker Augmentation, when Marker Augmentation is selected, the skeleton model walks normally.)

https://github.com/perfanalytics/pose2sim/assets/131518891/716d95d7-3e93-4cd6-b4ec-72d0e63fe8e6

- Can you also open your trc file in Excel, for example, to check if some keypoints are systematically missed?

I checked the trc file using excel as you've suggested, and so far, I could not see any patterns as to which columns are systematically having NaN values. It all seems random, albeit sequential in nature. (Am I right in assuming that the delimiter is tab?)

Here are some screenshots of the entire keypoints zoomed out in excel just to visualize where majority of the NaN values are

- Which pose estimation model did you use? Is it the same one as the one you specified in the Config.toml file?

We've used body_25 for our pose estimation model, this was also the option present in our Config.toml file.

_- You can also set interp_if_gap_smallerthan to 1000 or more, so that it interpolates even if there is a large gap.

I am assuming that the unit of this variable is frames. We haven't tried this yet but we will soon. I will update here once we've gotten the results.

_- Also, try to increase the value of reproj_error_threshold_triangulation and to decrease the one for likelihood_thresholdtriangulation.

We also haven't tried this step yet. Is the likelihood_threshold_triangulation the confidence value threshold that the program checks in all the 2D keypoints? What does reproj_error_threshold_triangulation do as well? We'll let you know once we've figured out the best value for these two.

Again, thank you so much for the help you've provided. We'll update soon regarding the last two steps we haven't tested.

[davidpagnon]

Thank you for your answer, indeed, it should not be jumpy like that. Let's keep tracking the potential issues down.

When we actually try to use the trc file with NaN values, OpenSim gives us an error and we aren't able to start the rendering process at all. Here is the error given by OpenSim.

It won't open in OpenSim, but you will have more success in the Blender add-on, if you have a chance to install it. This is what I use to debug my results: https://github.com/davidpagnon/Pose2Sim_Blender

_- You can also set interp_if_gap_smallerthan to 1000 or more, so that it interpolates even if there is a large gap.

This is frame, yes. I thought that it would not really make sense to interpolate beyond a certain gap width, so I set a limit; but it can be increased in the Config.toml file.

We also haven't tried this step yet. Is the likelihood_threshold_triangulation the confidence value threshold that the program checks in all the 2D keypoints? What does reproj_error_threshold_triangulation do as well? We'll let you know once we've figured out the best value for these two.

likelihood_threshold_triangulation: Exactly reproj_error_threshold_triangulation triangulates a point, reprojects it on each camera, and if it is too far from the original 2D point, it will try removing a camera and see if the error is better. If less than 2 cameras remain and the error is still larger than the threshold, the point is dropped.

It makes me think: one aspect to check might be the units of the calibration. If it is not in the right one, it may lead to artificially large errors.

Keep me updated, and if it still does not work you can send me some of your data so that I can test it!

[MeeroDLSU]

Good day!

Apologies for the delay, I have been recalibrating due to a misplaced Calib_scene.toml file. I have just finished recalibrating, and will now try to fine-tune the values.

It makes me think: one aspect to check might be the units of the calibration. If it is not in the right one, it may lead to artificially large errors.

Yes, we used meters for measuring the extrinsic 3D coordinates (we used "scene" for extrinsic calculations).

PS In the current updated main branch of pose2sim, I have also encountered the "broadcast error" similar to this issue: https://github.com/perfanalytics/pose2sim/issues/93

The fix of replacing imgp == [] with len(imgp) == 0 also worked for me

I will probably update within the day on the tuning results. Many thanks!

[MeeroDLSU]

Good Day!

So I have been working on this, attempting to get a bunch of points to show in the Blender_Pose2Sim. First thing that I was able to do was to finally get a trc file without any NaN values. I set likelihood_threshold_triangulation to 0.1 and reproj_error_threshold_triangulation to 50. I've also set the interp_if_gap_smaller_than to 50. So far, with these values, I was able to get a complete trc file that went through to Marker Augmentation.

After getting the trc file, I attempted to load this trc file into OpenSim, no luck here, the skeleton model clips through the floor when the IK file is loaded and run. I then installed the Blender_Pose2Sim inorder to visualize what is happening. Still no luck, when adding the trc file, there isn't any green points appearing in the screen as can be seen below.

I've only imported the markers since I do not have a camera.toml file configured and I just wanted to visualize the points first. Upon forwarding the playback to see if this is the same case for all frames, I am greeted again with an empty screen with no points in sight.

I am planning to keep on changing the values of the three variables mentioned, in hopes that it might fix this issue. What could possibly be the cause of this issue? Any ideas as to what I should try next?

Looking forward for your helpful response!

[davidpagnon]

Your results should not be bad, so there is definitely something. I keep thinking that calibration could be your issue.

• Are you sure you did not move the cameras between calibration and motion capture?
• What kind of camera did you use? If you used phones, did you take photos for the capture, or did you use a video? Taking photos for calibration can lead to wrong results, since the frame is often wider than for videos, even with standard settings.
• What did you calibrate on? The scene? A checkerboard? How far apart are the points? You can have decent results for calibration if the points are very close, but then out of this tiny space the triangulation results won't be that good.

What do you mean, you don't have a camera.toml file configured? You probably have a calibration somewhere, or you could not even run any triangulation!

[davidpagnon]

@MeeroDLSU if you can't figure it out, could you send me a link to your session folder? (here or per email at dp2032@bath.ac.uk)

[MeeroDLSU]

Good Day Sir David!

I've come bearing good news. I think I've got a good trc file, but before that let me address all of your points first.

Are you sure you did not move the cameras between calibration and motion capture?

Yes, we actually were mindful not to move the cameras when gathering data.

What kind of camera did you use? If you used phones, did you take photos for the capture, or did you use a video? Taking photos for calibration can lead to wrong results, since the frame is often wider than for videos, even with standard settings.

We used external webcams as our cameras. I was actually not aware of this. We took photos for the intrinsic checkerboard while we took frames for extrinsic calculations from the videos of each camera.

What did you calibrate on? The scene? A checkerboard? How far apart are the points? You can have decent results for calibration if the points are very close, but then out of this tiny space the triangulation results won't be that good.

We calibrated using the scene option. In total, we have 13 reference points. Upon calibrating, we've found out that including Points 1-6 results in a larger RMS.

We then tried omitting these points and just went with Points 7-13 to calibrate with and we've got a significantly smaller RMS calibration error (the first one I sent in this thread). I then used Points 7-13 to get the trc file. Upon reading this, I realized that our approach was wrong. I checked the blender file for the camera positions, and this is what it looks like.

This was not what our camera setup looked like, meaning that the calibration file was the problem all along. I then recalibrated, but this time I included all points in order to have a larger space. Next, I tried getting the trc file again with the same interpolation and thresholds params (complete step, from person association to marker augmentation). I was surprised that it actually works now. There is a humanoid figure of green dots in blender, and when I play the frames, the figure makes a walking move. It looks like this.

What do you mean, you don't have a camera.toml file configured? You probably have a calibration somewhere, or you could not even run any triangulation!

I am sorry for the confusion, I thought that Pose2Sim_cameras.toml file was another separate file. Turns out that it was just the Calib_scene.toml file all along.

Summary Overall, there seem to be values now albeit the calibration having a large RMS. When it comes to the output, the trc file seems to capture the overall movement of the participant. However, there is a weird movement at the very end of the animation. I recorded it and is attached below.

https://github.com/perfanalytics/pose2sim/assets/131518891/2d1b8923-4124-48a6-977a-4a2d7b3d1213

The whole trial is the participant walking from Point A to Point B and vice versa, there was no running whatsoever in the trial. My best guess as to what is causing the problem for the latter parts of the frames is due to the interpolated values (too large of a gap to interpolate, in that it doesn't make sense anymore). Also I forgot to setup framerate, which might be why everything is sped up.

[davidpagnon]

Awesome, thanks for the detailed answer!

The points you took from the checkerboard were not rotation invariant (odd number of columns and rows in your checkerboard points: it should be odd and even, or even and odd), and sometimes it results in weird flips of the camera. It seems like it was the case for the camera in the middle.

RMS is still rather large, but it is often better to have a slightly bigger error on a larger volume: otherwise, as soon as your subject is going away from the calibrated points, the triangulation results might be bad. Ultimately, the end results you need to watch are the triangulation error ones. You should choose the configuration that gives you the smallest triangulation error, even if the calibration error is larger.

It's hard to see the motion of the green dots on a smartphone screen, I should find a better way to represent them 😅 Before any further thinking, can you open the triangulated TRC file before marker augmentation and tell me if it does the same weird thing?

You can run Pose2Sim again with the correct framerate, it should not take long. Don't forget to set your target framerate in Blender as well when importing your markers.

[MeeroDLSU]

Hello again!

The points you took from the checkerboard were not rotation invariant (odd number of columns and rows in your checkerboard points: it should be odd and even, or even and odd), and sometimes it results in weird flips of the camera. It seems like it was the case for the camera in the middle.

I thought that this rotation invariant rule was only applied for intrinsic calculation. Does it also apply to extrinsic calculations as well? For our intrinsic calibration, I did find it weird that the numbers are reversed in order compared to your example.

Our Intrinsic Image:

Your example:

In my case, I just followed what the corner detection assigned when clicking the corners manually for images that could not be automated.

It's hard to see the motion of the green dots on a smartphone screen, I should find a better way to represent them 😅 Before any further thinking, can you open the triangulated TRC file before marker augmentation and tell me if it does the same weird thing?

No worries! I also tried visualizing the trc file into OpenSim since I haven't done the full installation for the Blender_Pose2Sim.

Here is the marker augmentation trc file in OpenSim:

https://github.com/perfanalytics/pose2sim/assets/131518891/5ee566d0-713d-4c54-87ed-615b67d34fc7

Here is the triangulation trc file (without filtering) in OpenSim:

https://github.com/perfanalytics/pose2sim/assets/131518891/fe9b4520-032d-41b0-9242-0fa21b8747f0

It seems that the issue is both present in the trc files. I don't know why it is happening, although I can assure you that the participant did not fly at the end, lol. Thank you again for your response. I am out of further ideas, except for adjusting the interpolation gap, which I heavily suspect is the cause of this. Let me know what you think!

[davidpagnon]

The rotation invariance is actually important for two things:

• the order of corner détection, but if you clicked points manually this should not be the problem
• camera calibration as well. I can't quite explain it but I've had weird results when calibrating with a symmetrical pattern. Sometimes it works, and sometimes the result is random

Your motion does not seem natural, so there is definitely something wrong. Maybe it's the accuracy of your calibration, whose error is still rather high. Are you sure that you measured the coordinates accurately enough? You should not have to change the Config.toml too much, unless you have faulty data.

I can see that your camera has slight lens distortions, and I did not find time to fully take them into account yet. However I still think that our main issue is due to calibration (and later on, maybe scaling but we'll check that later)

[MeeroDLSU]

Good day!

I also think that the main problem would be the calibration, due to its high error. I think the course of action that our group will take would be to remeasure points, and then try calibrating again. Unfortunately, we will not be able to do a recalibration until at least next week due to a semestral break.

Should I then close this issue and reopen a new one once we've calibrated again?

I can see that your camera has slight lens distortions, and I did not find time to fully take them into account yet.

Which image did you base this from? If it's based on the intrinsic checkerboard pic, then I think it might not be distortion but just the paper checkerboard not being flat enough (next time we'll laminate it instead so that it is entirely flat). But if it turns out that there is a distortion, what cameras would you recommend for us to purchase? The reason why we went with a webcam is due to our thesis trying to prove that clinics do not need expensive equipment to do gait analysis.

One more thing, I saw that keypoints calibration is in development. Would it be possible for me to contribute to this? Although my knowledge in camera calibration is very limited, I do have decent coding skills and a lot of free time from the sem break. I think when calibration through key points is developed, we will have an easier time calibrating.

Many thanks for your continued response!

[davidpagnon]

Should I then close this issue and reopen a new one once we've calibrated again?

Do what your heart says, I'm okay either way!

Which image did you base this from?

Just from the one you posted 2 messages above. It looks like the lines on the outside are slightly curvy, but honestly I don't think this is your problem.

One more thing, I saw that keypoints calibration is in development. Would it be possible for me to contribute to this?

On my side, of course I'm always happy for more support, thanks! @hunminkim98 volunteered for it, but lately he has been working on other stuff. He has got a good base but got stuck at some point, and I could not find time to really dive in the code with him (I work on Pose2sim as a side project, now, and am quite busy with my post-doc). Maybe this would be a good occasion to get back on the project? We can further discuss it with HunMin on our discord server: https://discord.com/invite/4mXUdSFjmt

[MeeroDLSU]

Got it! I'm closing this issue thread for now. I will be opening a new on one once we are able to recalibrate. Thank you so much for your helpful replies. Looking forward to work with you in these next few weeks!