How can I run pose2sim on my videos ?

[rutuja1409]

Hello,

I want to run pose2sim on some of my videos. I performed 2D pose estimation step. After that it says camera calibration, I'm not sure how to do this step ? Do we need cameras or just photos of checkerboard in a folder. Can we use the same .qca file present in the demo folder.

I added the checkboard images in calib-2d folder and ran Pose2Sim.calibrateCams(). My output - Residual (RMS) calibration errors for each camera are respectively [] px, which corresponds to [] mm.

After that Pose2Sim.track2D() gave me error return error_min, persons_and_cameras_combination UnboundLocalError: local variable 'persons_and_cameras_combination' referenced before assignment

Can you help me here @davidpagnon?

Thanks a lot!

[davidpagnon]

Dear @rutuja1409 ,

Thanks for reaching out! Calibration is not a trivial issue, and I'm planning to make it easier -- not sure when, though.

Calibration consists in two steps: calibrating parameters intrinsic to the camera (focal length, distortion, etc), and calibrating parameters extrinsic to it (rotation and translation regarding the center of the scene). As a consequence, the .qca.txt file I provided cannot be used for your own capture. You can calibrate intrinsic parameters of all cameras independently whenever you feel like it, and then calibrate all extrinsic parameters while on the scene. You will not be able to retrieved the latter after you put cameras back in their box.

The usual way is to calibrate intrinsics by taking about 20 pictures of a checkerboard from different angles, and to calibrate extrinsics either with the same checkerboard laid flat on the ground, or with a "wand" method.

Before I manage to find time to make calibration easier, here is what I suggest:

• Converting a calibration file from Qualisys with Pose2Sim.calibrateCams(). The easiest way, providing that you have Qualisys cameras (and that calibration has worked -- harder in broad daylight and if cameras are far away)
• Use a checkerboard with Pose2Sim.calibrateCams(). You can have a visual feedback to check whether the pattern is correctly detected of not
• Use AniposeLib calibration. I use the exact same folder structure and the same kind of output calibration file, so this should be seamless. This is probably much more robust than other solutions, so if accuracy is very important make sure to try it out
• Use Argus. I would actually recommend it, as going through the tutorials will make you have a better grasp of how calibration works.
• Alternatively, it is always possible to use the EasyMocap procedure, and then to convert the output yml files to the toml format Pose2Sim needs thanks to the Pose2Sim/Utilities/calib_yml_to_toml.py script.
• Or try multical. I haven't tried it yet, but this was the choice of another Pose2Sim user.

[davidpagnon]

I want to run pose2sim on some of my videos. I performed 2D pose estimation step. After that it says camera calibration, I'm not sure how to do this step ? Do we need cameras or just photos of checkerboard in a folder. Can we use the same .qca file present in the demo folder.

And to answer this more specifically, you need to take several pictures of the checkerboard per camera, one of them (typically the first or the last one) being used as a reference for extrinsic parameters.

calib-2d ├──calib_cam1_img -> about 20 pictures in each of these folders ├──... └──calib_camN_img

[rutuja1409]

Dear @rutuja1409 ,

Thanks for reaching out! Calibration is not a trivial issue, and I'm planning to make it easier -- not sure when, though.

Calibration consists in two steps: calibrating parameters intrinsic to the camera (focal length, distortion, etc), and calibrating parameters extrinsic to it (rotation and translation regarding the center of the scene). As a consequence, the .qca.txt file I provided cannot be used for your own capture. You can calibrate intrinsic parameters of all cameras independently whenever you feel like it, and then calibrate all extrinsic parameters while on the scene. You will not be able to retrieved the latter after you put cameras back in their box.

The usual way is to calibrate intrinsics by taking about 20 pictures of a checkerboard from different angles, and to calibrate extrinsics either with the same checkerboard laid flat on the ground, or with a "wand" method.

Before I manage to find time to make calibration easier, here is what I suggest:

• Converting a calibration file from Qualisys with Pose2Sim.calibrateCams(). The easiest way, providing that you have Qualisys cameras (and that calibration has worked -- harder in broad daylight and if cameras are far away)
• Use a checkerboard with Pose2Sim.calibrateCams(). You can have a visual feedback to check whether the pattern is correctly detected of not
• Use Anipose calibration. I use the exact same folder structure and the same kind of output calibration file, so this should be seamless. This is probably much more robust than other solutions, so if accuracy is very important make sure to try it out
• Try multical. I haven't tried it yet.
• Use Argus. I would actually recommend it, as going through the tutorials will make you have a better grasp of how calibration works.

Hello,

Thank you so much for your reply. I will follow the steps you suggested. And the error for Pose2Sim.track2D() was coming because calibration was not done right? or is it some other issue?

Thanks

[davidpagnon]

Exactly!

[carlosedubarreto]

@rutuja1409 , maybe I can have a sugestion to make you suffer little less. I made a tool to use easymocap multimacmera using blender. In that tool I made a workflow to make the calibration.

You can use this tool to get the intr.yml and extri.yml and use the pose2sim tool to convert (to get the toml file)

You can get it for free, there is a code on the description. https://carlosedubarreto.gumroad.com/l/ceb_easymocap_multicam

[rutuja1409]

@rutuja1409 , maybe I can have a sugestion to make you suffer little less. I made a tool to use easymocap multimacmera using blender. In that tool I made a workflow to make the calibration.

You can use this tool to get the intr.yml and extri.yml and use the pose2sim tool to convert (to get the toml file)

You can get it for free, there is a code on the description. https://carlosedubarreto.gumroad.com/l/ceb_easymocap_multicam

Hello Thank you for this easy tool. After we get intr.yml and extri.yml, we can convert them using pose2sim.calibrate() to .toml files ?

[carlosedubarreto]

Yeah @rutuja1409 . use this script https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_yml_to_toml.py

If I'm not wrong, it would be like this. For example, if the intri.yml and extri.yml are in the folder C:\calib

the path to the files would be C:\calib\intri.yml C:\calib\extri.yml

and considering that you want the toml file to be in the same folder C:\calib\Calib.toml

then, to use the calib_yml_to_toml.py script that is in the Uitlities folder, you will have to run like this

python calib_yml_to_toml.py -i C:\calib\intri.yml -e C:\calib\extri.yml -t C:\calib\Calib.toml

[carlosedubarreto]

Oh another important note. Since calibration is a confusing process (it took me months to undestand it) its possible that your calibration results in error using the pose2sim and opensim.

What I could suggest is try yo make a better calibration, this usually solves the issues I have.

[davidpagnon]

Lol thank you Carlos, I did not even have time to answer that you had already taken action!

[rutuja1409]

Oh another important note. Since calibration is a confusing process (it took me months to undestand it) its possible that your calibration results in error using the pose2sim and opensim.

What I could suggest is try yo make a better calibration, this usually solves the issues I have.

Okay. Thank you for all the steps. So how many videos would you recommend to me for better calibration for intrinsic and extrinsic parameters?

[carlosedubarreto]

Lol thank you Carlos, I did not even have time to answer that you had already taken action!

I know its not an easy thing to answer, and I already made a solution to try to make it easier. so... why not. it was the least I could do 😊

[carlosedubarreto]

Oh another important note. Since calibration is a confusing process (it took me months to undestand it) its possible that your calibration results in error using the pose2sim and opensim. What I could suggest is try yo make a better calibration, this usually solves the issues I have.

Okay. Thank you for all the steps. So how many videos would you recommend to me for better calibration for intrinsic and extrinsic parameters?

You came make it with at least 2 cameras. The important part for the calibration (if you are using the addon I suggested, is taking more than 20 frames of reference from the intrinsic calibration (the one that you record the chessboard in all possible positions from each camera.)

And the extrinsica calibration, I would suggest to do it manually, because my tests the automatica calibration is mostly faulty.

And when making the extrinsic calibration always start the marking from the same spot.

For example the image below

always start from the same spot, from all the cameras that you are going to use

when you have more cameras can be more confusing, but that is importante to pay attention.

on my youtube I have several videos showing about it https://www.youtube.com/c/CEBStudios/videos

You can see the videos about easymocp (not the monocular one) and mpp2sos. they might help you understand better this complicated part.

PS.: mpp2sos is another solution that I made on top of the one i suggested you.

[rutuja1409]

Oh another important note. Since calibration is a confusing process (it took me months to undestand it) its possible that your calibration results in error using the pose2sim and opensim. What I could suggest is try yo make a better calibration, this usually solves the issues I have.

Okay. Thank you for all the steps. So how many videos would you recommend to me for better calibration for intrinsic and extrinsic parameters?

You came make it with at least 2 cameras. The important part for the calibration (if you are using the addon I suggested, is taking more than 20 frames of reference from the intrinsic calibration (the one that you record the chessboard in all possible positions from each camera.)

And the extrinsica calibration, I would suggest to do it manually, because my tests the automatica calibration is mostly faulty.

And when making the extrinsic calibration always start the marking from the same spot.

For example the image below

always start from the same spot, from all the cameras that you are going to use

when you have more cameras can be more confusing, but that is importante to pay attention.

on my youtube I have several videos showing about it https://www.youtube.com/c/CEBStudios/videos

You can see the videos about easymocp (not the monocular one) and mpp2sos. they might help you understand better this complicated part.

PS.: mpp2sos is another solution that I made on top of the one i suggested you.

Hello, Sorry for many questions. But actually, I am new to calibration and it's a bit tough to understand. So while recording the video with the android phone for intrinsic we have to hold the checkerboard in our hands. But the camera should be fixed only in one position and it shouldn't move? So only we will move with the checkerboard.

And while recording the extrinsic the checkerboard is on the ground and then we can move the camera and record it?

Also I was following the steps below https://github.com/zju3dv/EasyMocap/blob/master/apps/calibration/Readme.md So do I need to install easy mocap or just downloading the project is enough?

[davidpagnon]

Don't worry, it is not trivial!

Intrinsic calibration can be carried out at any time. These parameters (focal length, lens distorsion, etc) will usually not change from one capture session to another. You can either move the checkerboard in front of the camera, or move the camera in front of the checkerboard.

Conversely, extrinsic calibration has to be done right before, or after, the capture session. These parameters represent the orientation (rotation and translation) of the cameras as regard to center of the scene, so moving cameras or the checkerboard (i.e., the reference frame) will mess any further 3D reconstruction.

About 20 images per camera are needed for intrinsic calibration, while only one will be used for extrinsic calibration.

I have never used EasyMocap myself, however for sure you don't need heavy machine learning packages such as torchvision, nor 3D mesh body models such as SMPL. Just using the scripts mentioned in the page you cite should be enough. To be honest, I am not sure that their procedure is easier than mine once you have understood calibration! If it becomes too complicated, make sure you check the other methods I quoted there, especially AniPoseLib which is rather light, robust, and provides the exact same calibration format as mine.

[rutuja1409]

b_cam1_im Hello,

Thank you soo much for in detail explanation. It helped me a lot. I was able to do the calibration with Aniposelib as well as with Pose2Sim.calibrateCams(). Calibration was successful and .toml was generated. However after running Pose2Sim.track2D() I'm still getting the error. Does this mean I have to add more images for calibration or is it some other issue ?

track_2d_all(config_dict) File "C:\Users\HRathi\AppData\Local\Programs\Python\Python38\lib\site-packages\Pose2Sim\track_2d.py", line 291, in track_2d_all error_min, persons_and_cameras_combination = best_persons_and_cameras_combination(config, json_files_f, personsIDs_comb, P, tracked_keypoint_id) File "C:\Users\HRathi\AppData\Local\Programs\Python\Python38\lib\site-packages\Pose2Sim\track_2d.py", line 174, in best_persons_and_cameras_combination return error_min, persons_and_cameras_combination UnboundLocalError: local variable 'persons_and_cameras_combination' referenced before assignment

[davidpagnon]

Awesome for the calibration!

Another error that I'll have to catch, to print a clearer message... This means that the variable 'persons_and_cameras_combination' has not been defined, and yet you want to return it. This variable is defined in a loop (line 117 or 119), which you probably haven't entered. To be more specific, there is either no available persons in your json files, or that there is no json file altogether. Or that your Calibration file is empty.

Can you send me your Calib file? Also, are you sure you have populated your pose-2d folder with as many subfolders as you have cameras, and that these camera folders contain your OpenPose json files?

[rutuja1409]

Here is my calibration file generated from both Aniposelib as well as Pose2Sim.calibrate. Converted it to notepad and attached. And before calibration I have done 2d pose estimation with openpose. I captured the video of person cycling and so pose 2d just has 1 cam folder which has json files. Do we have to capture our dataset videos with all cameras which we used for calibration ? I just captured the dataset video with 1 camera and used 3 cameras for calibration.

Calib.txt Calib1.txt

[davidpagnon]

Okay, so two things remain unclear to me.

• There are 3 cameras in your calibration file: why do you only have one cam folder for you person cycling?
• The calibration files cannot be right, since they are so dissimilar from each other.
  • First, intrinsic parameters, camera matrix in particular. Did you take about 20 pictures of the checkerboard from each camera? \ Was the checkerboard close enough that corners could be correcly detected? Set 'show_corner_detection' to true in the Config.toml file to make it sure.
  • Then, extrinsic parameters, i.e., rotations and translations. In both cases, your cameras are looking away from the center. With AniPose, cameras are grouped together, while with Pose2Sim, they are not. Note that translations are much bigger with Pose2Sim, but this is not a problem: it works in meters, while anipose uses millimeters.\ Are you sure that the checkerboard was visible by all cameras on the frame you used for reference?

In any case, if you only have 3 cameras, you should make sure you decrease the 'min_cameras_for_triangulation' parameter to 2 in Config.toml.

[rutuja1409]

Okay, so two things remain unclear to me.

• There are 3 cameras in your calibration file: why do you only have one cam folder for you person cycling?

• The calibration files cannot be right, since they are so dissimilar from each other.

  • First, intrinsic parameters, camera matrix in particular. Did you take about 20 pictures of the checkerboard from each camera? _Was the checkerboard close enough that corners could be correcly detected? Set 'show_cornerdetection' to true in the Config.toml file to make it sure.
  • Then, extrinsic parameters, i.e., rotations and translations. In both cases, your cameras are looking away from the center. With AniPose, cameras are grouped together, while with Pose2Sim, they are not. Note that translations are much bigger with Pose2Sim, but this is not a problem: it works in meters, while anipose uses millimeters. Are you sure that the checkerboard was visible by all cameras on the frame you used for reference?

In any case, if you only have 3 cameras, you should make sure you decrease the 'min_cameras_for_triangulation' parameter to 2 in Config.toml.

Hello @davidpagnon Sorry for late reply, I was figuring out stuff.

I started from beginning with new data. I took 3 camera. Shoot dataset videos and camera calibration videos. All the steps are done without any error till Pose2Sim.filter3D()

Now I loaded the model in Opensim. While loading the scaling model, the run button doesn't get activated. So I was not able to do scaling. Here is the drive link for calibration and video dataset and also included the .trc file that got generated. https://drive.google.com/drive/folders/1JlS4frirBzir4Fgd_gM1F0KpYoiVQsoZ?usp=share_link

Can you please help me here, I'm almost on my last step.

Thanks, Rutuja

[davidpagnon]

Hello Rutuja,

Unfortunately I will be completely swamped until the 8th of December and won't be able to help... Just an idea, can you try to shut and reopen the scale tool? Sometimes OpenSim suffers from weird bugs. If it is not the problem, I'll see to it in a couple weeks :)

[rutuja1409]

Hello Rutuja,

Unfortunately I will be completely swamped until the 8th of December and won't be able to help... Just an idea, can you try to shut and reopen the scale tool? Sometimes OpenSim suffers from weird bugs. If it is not the problem, I'll see to it in a couple weeks :)

Hello, Okay, no issues, whenever you have time. Just a question, while loading the .trc file for scaling, In the static pose weight tab I get the following error From-File not found? Does this happen because of any specific reason like calibration or anything?

[davidpagnon]

We don't have to adjust the positions of model markers, so we don't need to mess with this pane. OpenPose is assumed to estimate joint centers with a systematic bias, which is similar regardless of the subject, and not dependent on any operator inaccuracies.

In any case, this is mostly useful for adjusting the positions of clusters, which cannot be positioned very accurately since they are not placed on any specific body landmarks.

So to be clear, whatever your error is, it shouldn't be it. Last check, did you keep the "adjust model markers" box unchecked?

[rutuja1409]

We don't have to adjust the positions of model markers, so we don't need to mess with this pane. OpenPose is assumed to estimate joint centers with a systematic bias, which is similar regardless of the subject, and not dependent on any operator inaccuracies.

In any case, this is mostly useful for adjusting the positions of clusters, which cannot be positioned very accurately since they are not placed on any specific body landmarks.

So to be clear, whatever your error is, it shouldn't be it. Last check, did you keep the "adjust model markers" box unchecked?

I just load the Scaling_Setup_Pose2Sim_Body25.xml file and then load my data .trc file which is in pose3d. After that I saw this red error on static pose weight tab. I don't know if its the error or not and how it generated . I have checked the adjust marker model. Let me know how to solve it after you see my drive link with calib and .trc file, whenever you get time.

[davidpagnon]

Leave the "adjust model markers" box unchecked (as it should be when you load the setup file), and tell me what it does.

Also, you can check whether the triangulation worked by loading your trc file in OpenSim. I don't have access to it right now, but it should be something like File - > Preview experimental data

[rutuja1409]

Leave the "adjust model markers" box unchecked (as it should be when you load the setup file), and tell me what it does.

Also, you can check whether the triangulation worked by loading your trc file in OpenSim. I don't have access to it right now, but it should be something like File - > Preview experimental data

Yes unchecked "adjust model markers" still the run button is not activated and loaded the .trc file by File - > Preview experimental data. This is what I got.

[davidpagnon]

Okay, I'm sorry, I'll have to check your data later then :/ Ping me if I forget after the 8th!

(try to zoom out from the preview and to go forth a few frames, and if the marker movements don't seem realistic, it means that the problem comes from an earlier step)

[davidpagnon]

Hi @rutuja1409, I'm finally finding some time to spare on pose2sim, did you manage to get a little further?

I just downloaded your files, and there is definitely a problem with your calibration:

• First, for the intrinsic parameters, you should put your checkerboard on a rigid board (cardboard, hard plastic...)
• Then, and most importantly, the extrinsic parameters characterize the position and orientation of the camera. Hence, from the moment you start calculating them, your cameras need to stop moving at all.

In other words, your cameras need to be priorly set on a solid stand / tripod. Only once it is done, you can start to determine extrinsic parameters. The cameras still need to stay on the stand when you perform your arm movements.

PS: Also, OpenPose works best if the whole body is visible.

[davidpagnon]

This issue has been marked as stale. Don't hesitate to reopen it if you need further help!