Errror with markertrajectory

[rlpjanssen]

Hi!

I was converting video images from 3 cameras and following the steps:

from Pose2Sim import Pose2Sim Pose2Sim.calibrateCams() Pose2Sim.track2D() Pose2Sim.triangulate3D() Pose2Sim.filter3D()

The .trc files i get after filtering are no good to use in OpenSim. I added images of the failed markers:

Has anyone also had this issue? I could really use some help. Thanks in advance! Kind regards, RLP Janssen

[davidpagnon]

Well there is definitely a problem at some point, since you've got no output. It's difficult to tell with that little information, but the usual issue is related to calibration. I really plan to make a tool for making it easier (there is one in the "draft" branch of the repository, but it is not ready for production yet).

First question, how do you calibrate?

[rlpjanssen]

i calibrate with a checkerboard. indeed the calibration is not really good: i had to put the reprojection error value high due to the fact that else i got nan values

[davidpagnon]

Yes, calibration is definitely the most challenging part here. I have to admit that the tool I provided is not as user-friendly, robust and flexible as I initially thought. This is a pain since you cannot do anything without a good calibration. However, the good news is that once calibration is fixed, everything else is straightforward.

While I plan to work on this relatively soon, I cannot guarantee I'll easily find time in the next few weeks. I would like to provide a few robust and easy methods (if you would like to get involved, check the to-do list in Readme: there are a few references of methods I would like to explore). In the meantime, you can use this script:

https://github.com/perfanalytics/pose2sim/blob/draft/Pose2Sim/Utilities/calibrate_extrinsics.py

Here is how it works:

• First, keep aside the calibration file that you obtained with calibrateCams(). The extrinsic parameters are probably wrong, but the intrinsics should be fine. At least, provided that you used enough views of your checkerboard (about 20) from different angles, and that corners are correctly detected (you can verify this by setting show_corner_detection to true in the Config.toml file).

• Put a large checkerboard down, or just take the coordinates of anything in the scene, be it lines on the floor, a frame, a table, anything that you can get 3D coordinates of. Try to gather 10-30 points, covering as much space as possible. You will write those coordinates in the objp matrix of calibrate_extrinsics.py.

• Extract an image from each video viewpoint, for example with this script:

  import cv2
  vidcap = cv2.VideoCapture(`C:\data\experiment\Empty_project\calib-2d\calib_cam1.mp4`)
  success, image = vidcap.read()
  if success:
  cv2.imwrite("extrinsics_cam1.jpg", image)  

• Edit the variables in section 1. If your calibration file is C:\data\experiment\Empty_project\calib-2d\Calib.toml and the images used for extrinsic calibration are named like C:\data\experiment\Empty_project\calib-2d\extrinsics_cam1.jpg, then:

  folder = r'C:\data\experiment\Empty_project\calib-2d\'
  calib_file = os.path.join(folder, r'Calib.toml')
  img_root = 'extrinsics_cam'
  cam_nb = 1 # reiterate with 2, 3, etc for each camera

• Follow instructions in the file. Run each element by copy-pasting it in a python terminal:

  • go to the folder where it is downloaded, SHIFT + RIGHT CLICK in an empty space, "open a terminal here",
  • conda activate Pose2Sim
  • ipython
  • copy paste line 1 to 53 It will open an image, you will have to click your points, and hit ESC once you are done. A list will be output in the terminal, that you will have to copy paste in the imgp line of the camera you are considering. N.B.: if there are some object points that are not visible, remove them for this camera in the objp matrix.
  • copy paste imgp and objp in the terminal

• Section 2:

  • replace mtx and dist by the matrix and distortion found in your calib.toml file
  • Copy-paste this section into your terminal The script will match the clicked image points with the measured object 3D points, take into account the intrinsic parameters, and give you extrinsic parameters that you can copy-paste in the calibration file.

• The rest of the script is dedicated to verify that the reprojection error is low enough, and can be executed once you have calculated extrinsic parameters for all cameras.

[rlpjanssen]

Hi! Thanks a lot!

How do i exaclty obtain the coordinates of on object in the scene? Can i take the checkerboard from the calibration videos?

And do i then need to take this object from each camera view? I have 3 cameras.

This was my calibration by the way: I thought it looked quiet allright here but furtheron with the triangulate i thus had problems.

[davidpagnon]

You can use any object, so a checkerboard would work. Ideally, it should be as large as possible, and contain at least 10-15 points. Don't forget that you need to write down its 3D coordinates, so in the case of a flat object such as a checkerboard, don't forget to add a 3rd coordinate that you will set to zero.

For example, if you have a checkerboard with 3 x 4 20 cm squares, your objp would be: objp = np.float32([[0,0,0], [0,0.2,0], [0,0.4,0], [0,0.6,0], [0.2,0,0], [0.2,0.2,0], [0.2,0.4,0], [0.2,0.6,0], [0.4,0,0], [0.4,0.2,0], [0.4,0.4,0], [0.4,0.6,0]])

And then you would use the script to click and retrieve the points on each camera view and obtain extrinsic parameters, one by one.

I am not sure I see any obvious mistakes in your calibration values, but I have a few questions:

• Did you check whether the corner detection was good?
• I can see that one camera is in landscape, and two in portrait. Was it actually the case, or could the mistake lay there?
• Your ret value is quite high in the 2nd camera (it should be below 1), and fu and fv are very different. Can you check this one in particular?
• Also, I don't remember what exactly but I made some changes in the calibrateCams() function that I may have not reflected in the script that you used in the standalone utility script. I would advise you to try again with the regular Pose2Sim.calibrateCams() procedure and the Config.toml file.

[rlpjanssen]

The corner detection is indeed not good at all... i didn't notice this before. I get a load of 'Corners not found.' It is in fact the case that 2 cameras were vertical and one horizontal (2 iphones and one iPad). I don't know why the corners aren't detected tho? The checkerboard is in fully visable in all frames.

my config file looks like this: Our calibration checkerboard in the videos look like this: Do we need a white boarder around the checkerboard?

Thanks a lot already for helping me out!

[davidpagnon]

You're welcome! I did realize that the opencv function I used to detect corners is not very robust. I was actually thinking of replacing it with a simple neural network, but again, that's a simple project that still requires a little bit of time. However, it works quite a lot better if there is a visible white margin around the checkerboard (such as on this page: https://calib.io/pages/camera-calibration-pattern-generator). I would advise to tape the paper on the other side, so that the limit between white and black is even clearer. And you're welcome :)

[rlpjanssen]

Hi @davidpagnon, Thanks for all the help! The calibration is going way better now, all the corners are found! Does this look like a normal calibration to you? We have some problems at the triangulation now: not all marker trajectories are scaled the same way. This is for example the knee and the hip: This can't be right? Thanks in advance!

[davidpagnon]

Hello, your calibration does not stand out as problematic, but it is hard to say without the file. I can see that now, all cameras are in landscape mode?

For me to be able to better help you, do you think you could send me a minimal working example, with your calibration file, a few json files for each camera, and potentially your 3D file too? Also, are you sure that your cameras are synchronized? How do you handle this?

[rlpjanssen]

Hi,

I edited the videos again: all the cameras are vertical now and not horizontal. The calibration looks like:

I synchronized the cameras in Davinci Resolve based on waveforms. I will send you an e-mail with some footage and files!

[davidpagnon]

Thank you for sending me the documents!

So the problem is definitely the calibration step.

• Intrinsics: your results should be consistent, but it would be more exact if you tried to go more in the corners, and to position the checkerboard at an angle more often than frontally to the camera plane.
• Extrinsics: this is where I think the main issue lies. What you need to remember is that the last image is used for setting the coordinate system. Consequently, the same image needs to be seen by all cameras. From the videos you sent me, I don't think this is the case.

Hang in there, calibration is complicated, but then everything should come together smoothly! (and again, I plan to code an easier process, it's just hard to find the time :'( )

[rlpjanssen]

Hi David!

Thanks for all the help! We got it working now! The calibration is indeed a bit tricky, so thanks for all the tips!

Le mar. 11 avr. 2023 à 11:12, David PAGNON @.***> a écrit :

Thank you for sending me the documents!

So the problem is definitely the calibration step.

• Intrinsics: your results should be consistent, but it would be more exact if you tried to go more in the corners, and to position the checkerboard at an angle more often than frontally to the camera plane.
• Extrinsics: this is where I think the main issue lies. What you need to remember is that the last image is used for setting the coordinate system. Consequently, the same image needs to be seen by all cameras. From the videos you sent me, I don't think this is the case.

Hang in there, calibration is complicated, but then everything should come together smoothly! (and again, I plan to code an easier process, it's just hard to find the time :'( )

— Reply to this email directly, view it on GitHub https://github.com/perfanalytics/pose2sim/issues/23#issuecomment-1502964245, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4J76F42RSBYSBDBUYC22M3XAUOBNANCNFSM6AAAAAAWQGE3IE . You are receiving this because you authored the thread.Message ID: @.***>

[davidpagnon]

Awesome, I'm glad it helped! Did it all work all the way to the inverse kinematic step? If so, do you want to close the issue?

[rlpjanssen]

Yes, it worked all the way! Thank you! It can be closed.

Le mer. 12 avr. 2023 à 12:00, David PAGNON @.***> a écrit :

Awesome, I'm glad it helped! Did it all work all the way to the inverse kinematic step? If so, do you want to close the issue?

— Reply to this email directly, view it on GitHub https://github.com/perfanalytics/pose2sim/issues/23#issuecomment-1504999303, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4J76FZYPXNAXXAWJEMZ5LTXAZ4KLANCNFSM6AAAAAAWQGE3IE . You are receiving this because you authored the thread.Message ID: @.***>

[davidpagnon]

Just in case, I finally released a tool for a robust and simple calibration. If you ever end up using it, please tell me if anything goes wrong or is not clear!