Wrong Calibration file

[ClarkGrif]

Hello,

First of all, thanks for sharing your work with us!

I am trying to follow the steps mentioned in the README file, and I have reached the camera calibration. However, my calibration file is a .cal because I am using Optitrack PrimeColor cameras, which can be calibrated and synchronized through Optitrack Motive. How can I make it work?

Thanks, Clara

[davidpagnon]

Hello, I don't know these files, can you send me a sample so that I can see if I can translate it to the OpenCV format?

[ClarkGrif]

I cannot send it to you because the file type is not recognized. Do you have an email or any other means through which I can send it to you?

[davidpagnon]

Sure, you can send it to contact@david-pagnon.com, or preferably if it is not a binary file, you can simply copy-paste its content here.

[ClarkGrif]

Thank you for your answer, I have sent it by email.

[davidpagnon]

I received it, thanks! Unfortunately, this is a binary file and I cannot read it. Is there an option to export it as a txt file? You may need to contact your support center if this is not natively possible, they might be able to do something about it.

[ClarkGrif]

I contacted the support center to see if they can do something about it. Thank you !

[claraaudap]

Hello,

I gace up with the calibration file .cal because they cannot or don't want to give me an other type of calibration file so I want to do the calibration with a chessboard.

I followed the steps mentioned in the README file, and I have reached the camera calibration. I'm trying to do the calibration with the chessboard, so I took videos of a checkerboard with the 8 cameras I have and put them in a different folder for each camera but when I run the following script : Pose2Sim.calibrateCams() an error appears :

I don't understand what I'm doing wrong Can someone help me

Thank you, Clara

PS : Attached, you will see an example of the chessboard video

[claraaudap]

Attached, you will find my config file

Config1.txt

[davidpagnon]

Hi, this is a shame because converting a calibration would have been much easier than doing the calibration yourself...

I cannot see the attached example of the chessboard video. If you want me to test it, you can send me your whole project folder (my email address: contact@david-pagnon.com).

From the error you sent, it seems like no images are found, which could be caused by

• A wrong video extension name (beware of capital letters)
• A wrong folder name
• Not enough frames in the video

Not sure this is connected, but if I were you I would name my projet folder something different from "Pose2Sim" since it is the exact name of the package: it could confuse the system and make it look for files in the wrong folder.

Also, and although this should not have anything to do with the error, I would recommend calibrating with images rather than with videos. This way, you can choose the best ones for calibration. You can extract frames from a video with ffmpeg for example.

[ClarkGrif]

Thank you for your answer. I have sent you my file by email. As for me, I am trying what you advised me.

[davidpagnon]

Thank you Clara, I got the files.

There are 2 minor issues with your Config.toml file:

• First, vid_extension should be 'avi' rather than 'AVI' (at least on Linux and Mac, it is case insensitive on Windows)
• Second, corners_nb should be the internal corners: in your case, [6,8] rather than [7,9]

The way you do it, intrinsic parameters will be pretty well estimated, at least as long as you use 10 or more checkerboard images for each camera. These correspond to the camera properties, and usually need to be calculated only once. However, you also need to obtain extrinsic parameters, i.e. the position of all cameras as regards to your reference frame. This frame of reference can be set by putting the checkerboard down, in a way that it is visible to all cameras. In Config.toml, you can choose frame_for_origin so that it specifies whether the checkerboard is in place in the first or the last image.

Calibration should work with your checkerboard since it is relatively large, but I had more accurate results with using other more spaced reference points in the room. See this issue if you want to try out this method.

[ClarkGrif]

Thank you David for your help.

First, I'm gonna try with my checkerboard and after try the other method you gave me.

But I just didn't understang how to obtain extrinsic parameters. Do I need to take an other video with the checkerboard down in a way that it is visible to all cameras ? Or, am I just laying the checkerboard in a way that it is visible to all cameras and run calibrateCams() ?

[claraaudap]

When I run calibrateCams(), this message appears :

And this picture appears :

[davidpagnon]

Okay, perfect! To keep detecting corners on the next images, you can just close them one by one. This setting is mostly set to make sure corners are correctly detected, but it is annoying to close windows one by one. To disable it, you can set "show_corner_detection" to false in Config.toml

[ClarkGrif]

Will it pass by camera 2 on its own?

[davidpagnon]

But I just didn't understang how to obtain extrinsic parameters. Do I need to take an other video with the checkerboard down in a way that it is visible to all cameras ? Or, am I just laying the checkerboard in a way that it is visible to all cameras and run calibrateCams() ?

Hi Clark, No, only one video per camera is enough for both intrinsic and extrinsic calibration.

I would recommend filming with all cameras together, showing the checkerboard to each of them carefully (exactly the way you already did), and then stepping back and laying it down to set your reference frame (i.e., your extrinisic parameters). Alternatively, you can do the opposite: first lay the checkerboard down, and then show it to each of your cameras. In this case, frame_for_origin should be set to 0 instead of -1.

It will do all cameras on its own.

[claraaudap]

Great, thank you I get it ! And last thing, what does it means :

I'm sorry for all this questions !

[davidpagnon]

Oh, so this is a different problem! It seems like a numpy function I was using just got deprecated. I just solved it, could you type pip install pose2sim --upgrade to implement the change?

(Sorry this is just curiosity, but am I answering to two people, or to one person with two different accounts? 😅)

[claraaudap]

Sorry ahah, you are answering to one people with two accounts !

So I upgrade pose2sim and the error disappears!

[davidpagnon]

Awesome! Tell me if you experience any other errors, especially with extrinsic parameters.

[claraaudap]

Hello,

I tried to film with all cameras together, showing the checkerboard to each of them carefully and then stepping back and laying it down but I think it's not working well. I sent you an email with an example of my video. Can you check what I am doing wrong please. And is it normal that the calibration takes a lot of time for a 3minutes video ?

Thanks Clara

[davidpagnon]

Hi Clara,

First of all, your coverage of the camera field by the checkerboard is good.

Then, it takes long because you optimize calibration over a ton of detected points. Considering the number of points in your checkerboard, I would say you need between 10, and maximum 30 images of the checkerboard. There are two solutions to it:

• either you increase the vid_snapshot_every_N_frames parameter,
• or better, you extract all images from each video (with ffmpeg, for example) and carefully choose a few of them on which to do the corner detection.

This is tedious, but this has to be done only once for each kind of camera. Actually, are they all the same kind, with the same lenses? If so, check the Calib.toml file you obtained: are all intrinsic parameters (especially matrix) roughly the same? If so, you don't need to do it again, intrinsic calibration has worked.

Then there is extrinsic calibration. What I suspect is that the corner #1 in each video does not correspond to the same one on the checkerboard, when it is laid on the floor. Can you check this by setting show_corner_detection to true? This is the problem when cameras are all around the subject, and not in the same semi-circle (and this is why I should implement the possibility to work with a Charuco board, but that's another subject).

There is another option for extrinsics though, which is to use the temporary script I wrote and explained there: https://github.com/perfanalytics/pose2sim/issues/23#issuecomment-1493291952 The idea is to point by hand each point you want to use for calibrating extrinsic (whether it is the checkerboard or anything in the environment), in the right order.

[ClarkGrif]

The problem is that when the checkerboard is on the ground, the cameras do not detect the corners. And for the second option, I didn't understand what I'm doing with the script, where I put it ?

[claraaudap]

I finished the calibration and this is the file I've got

[claraaudap]

Do you think the data are good ?

[claraaudap]

And second question, is it normal to have 'nan px' for the 2D tracking of person ?

[davidpagnon]

Okay, so there are several things:

• Intrinsic parameters don't jump to me as inconsistent: the "matrix" results are pretty close from camera to camera. I would tend to trust them

• Extrinsic parameters can't be good if corners are not detected when the checkerboard is on the ground, so the rest of Pose2Sim cannot work. I agree this is really a pain, however the good news is that once calibration is fixed, everything else is straightforward.

I would really advise you to use the script I linked there. The main difference is that in the original function, extrinsic parameters are calculated from the detected corners on the last image; with the proposed script, they would be calculated from points you would point by hand (see image for example points you could label). Have a look, I edited the post to give more details about how to use the script.

[ClarkGrif]

The above calibration file was generated for another video. On the other video, it detects the corners well. So do you think the data of the calibration are good? I'm gonna try with the script.

And for the 2D tracking of person, is it normal to have 'nan px' ?

[davidpagnon]

No, it's not normal to have Nan px nor to have to exclude the maximum number of cameras on every frame. It means that it cannot find a correct triangulation of your neck point.

There are 2 questions:

• does it manage to find corners when the checkerboard is laying on the floor?
• if it does, are corners detected in the same order? It is probably not the case since your cameras are around the subject rather than in the same semi circle, which is why I think you should use the script for extrinsic parameters.

I just managed to find a little time to work on making an easier calibration solution this morning, but I won't be able to work on it everyday so I would not expect it to be ready before mid-July.

Two other options if you can't make it work:

• try again to demand Optitrack the possibility of exporting the calibration to text file
• calibrate with the EasyMocap tool and convert it with this: https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_yml_to_toml.py

[ClarkGrif]

Thnk you for your time ! I'm gonna try what you gave me

[davidpagnon]

I don't know why I cannot see your last messages here but I got them per email, so I am going to answer what I can anyway. Can you see them?

• first of all, the error in calibrate_extrinsics: I changed the name of a variable to make it clearer, but I forgot an occurrence... Hence the non-defined variable, sorry I just fixed it!
• it is actually always best to have the cameras around the person, the only thing is that it does not work with the current calibration procedure 😅. Hence the other script, which is more versatile. Can you give me the output of calibration and track2D? And can you also attach the calibration file? I suspect that your cameras are too much in line. With the current calibration procedure, cameras cannot form over a semi circle, but they cannot be triangulated either if they are too close to a line.

[ClarkGrif]

I have a question : How to distinguish good calibration from bad calibration? Because I just finished setting up Pose2sim with 3 aligned cameras so that they all detect the checkerboard in the same direction. However, what I visualized in OpenSim had nothing to do with the recorded video. So, I was wondering if it could be due to the calibration or some other parameter.

[ClarkGrif]

I don't know why, me neither, I cannot see my previous messages. If you want I can send you by mail my folder of th project, it can be easier.

[davidpagnon]

Okay why not, send me your folder. I may not be able to work on it before next week, though.

There should be an output in the terminal that tells you about the calibration residual error (you can also find it in the log file that should be in your project folder). A good error is below a centimeter (2 or 3 cm should still give you coherent results)

[ClarkGrif]

Thank you!

I just checked and the résidual calibration errors are [3.357 , 4.672 , 4.069] mm. So the results should be coherent ! But for the Tracking of person, the mean reprojection error is 28.5px. So maybe it comes from there. I don't know. And for the triangulation, i've got this :

[ClarkGrif]

Hello, I just tried to calibrate using the script, and an error occurs when I do the 2D tracking of the person.

I have several questions regarding the script. When I write down its 3D coordinates of my checkerboard, 6x9 80 mm squares, should I write them in mm or m? And when I click point by point, should I do it in rows or columns? Does it matter?

Thank you

[davidpagnon]

Hello Clara,

I am sorry I am not very available these days. Among other projects, I am actually trying to make a better calibration tool that I would really like to release around mid-July (no promise, unfortunately).

Results are not completely incoherent (open your resulting TRC file with OpenSim: File -> Preview experimental results). But they are definitely not good enough for biomechanics. It is not normal that mean reprojection error is above the thresholds.

• Setting the cameras higher is a good idea to make sure the checkerboard is correctly detected when laid on the ground, however then the keypoint detection is not as good (you can use one of my Utility scripts there to verify this). But I lowered the likelihood_threshold and results were not much better, so this is not the main problem

• The main issue is that one or more cameras have to be excluded every frame to meet the conditions specified in Config.toml. When you only use 3, you should expect to not have to exclude any. Of course, using more is recommended and would lead to better results, but still, it shows that there is a calibration issue. I tried tweaking parameters such as likelihood_threshold and reprojection_error_threshold, however it did not make a large difference

• I would tend to trust the intrinsic calibration results, although they are probably not perfect since the "matrix" is not the same despite calibrating cameras from the same model

• So the issue is extrinsic parameters. There are two potential explanations:

  • either the checkerboard is still not accurately detected when laid on the ground,
  • or this represents a too small area, which leads to good results just where it is laid, but not as good further from it

[davidpagnon]

Hello, I just tried to calibrate using the script, and an error occurs when I do the 2D tracking of the person.

I have several questions regarding the script. When I write down its 3D coordinates of my checkerboard, 6x9 80 mm squares, should I write them in mm or m? And when I click point by point, should I do it in rows or columns? Does it matter?

Thank you

The unit does not really matter, although it is more consistent to use the same as the square_size one (mm in your case). And you should click the image points in the same order as you defined the object points.

If your object points are like np.float32([[0,0,0], [80,0,0], [160,0,0], [80,0,0],[80,80,0],[80,160,0]]) Then your image points need to be the origin, one corner on the right, two corners on the right, and same thing one row above.

[claraaudap]

Thank you for your response. It'q fine if you cannot answers right away. It's already really helpful and nice to take time to read and reply all the time.

I just share with you my result. If you want to comment you can.

When I use the script. I get this calibration result :

I don't know if it looks fine.

But when I'm doing the 2D tracking of person; I still have "nan px" and a warning, I don't know where it comes from and I'm pretty sure it's not what I need to have.

For the triangulation, I have the same warning then for the tracking. And got this result

And for the filter3D, I have this kind of graph :

All of this, give me something really not workable on Opensim

[davidpagnon]

If track2D does not work, nothing else will. And in this case, it is most likely a calibration issue usually on extrinsic parameters.

• Are you sure your objp coordinates are good?
• Are you sure you clicked them in the right order in the image?
• Are all objp visible in image? If not, remove the ones that cannot be seen for the concerned images.
• Do they take a large enough space on the camera frame?

I have the feeling that it would be much easier to talk through it by visioconference. I am not going to be available this week, but next week I'll be rather flexible.

[ClarkGrif]

I mean I'm sure I'm doing something wrong but I don't know why ! It would be great to do a visio to talk about it. Next week is perfect for me !

[claraaudap]

I can explain you how I use the script.

First, I write the coordinates in the objp matrix of my checkerboard. As I said it's a 6x9 80 mm squares :

I extract from all my cameras and click on every point like this :

After I replace the matrix and distortion found in my calib.toml file which give me :

Finally, I put this in the terninal and I've got this result :

This result looks really weird because all the data are kinf of the same but the cameras are in different position.

[ClarkGrif]

I found my mistake !! The result is still not workable but it's better.

[davidpagnon]

Solved after visioconference. The problem was the definition of the object points, a few of them were wrongly defined.

On another note, there may be an easier solution with Optitrack.

• They do not provide intrinsic properties, but these can be calculated with Pose2Sim.calibrateCams()
• They do provide a C++ API to retrieve extrinsic properties. See TT_CameraXLocation, TT_CameraYLocation, TT_CameraZLocation, TT_CameraOrientationMatrix. Translation can be taken as is, and the 3x3 rotation matrix can be converted to a Rodrigues vector with OpenCV.

Please keep me updated if there is anything new!

[davidpagnon]

Actually @claraaudap @ClarkGrif , if you manage to make it work with Optitrack, would you mind posting a quick tutorial in a new issue so that I can add it to the documentation? 🙏

[ClarkGrif]

If it's working, I'll do that ! Thank you for your help with all my issues

[ClarkGrif]

Hello, just a last question, what is the difference between OpenPose and the track 2D ?

[davidpagnon]

• OpenPose detects the pose on videos,
• track2D chooses the right person to triangulate in case several are detected by openpose (otherwise person0 can correspond to different persons across cameras, which leads to non consistent triangulation)