List index out of range

[peterlololsss]

Hi there! Currently I'm at V1.95, and during running steps after poseEstimation,(i.e. synchronization(), personAssociation(), triangulation(), markerAugmentation() ) I got no other than Error: List index out of range.

At first I thought it was due to high calibration error, (log showing Residual (RMS) calibration errors for each camera are respectively [2.219, 1.948, 2.032, 2.019] px, which corresponds to [12.057, 11.336, 10.647, 10.072] mm.) However, I recalculated by using easymocap this time, with error rate around 0.2 px, the error persist.

I also tried :

1. Reshoot the video from the range of 20s to 10s .
2. Change the folder structure from session to a single trial.
3. Modify the config file by increasing the error threshold by 10 times.

They all failed unfortunately while list index out range shouting at me:(

Looking forward to your reply! Thx for reading the issue:)

[davidpagnon]

Hi, at what stage do your get the error? Can you copy paste the whole message?

A few thoughts in the meantime:

• did you reinstall a conda environnement with python 3.9+ ? Due to the new libraries involved, 3.8 is no longer supported.
• I would not compare too much results from Easymocap and Pose2Sim calibration, the process is a bit different and a larger reprojection error might not coincide with worse triangulation results. Out of interest, did you use a checkerboard with Easymocap? Did you use the same checkerboard with Pose2Sim, or did you use elements from the scene?
• batch processing is currently broken, It'll try to find time to fix it but it's a bit hard these days

[peterlololsss]

Hi, After I successfully calibrate and poseEstimate(the pose folder is there with json files), I got the error from [synchronization, personassociation and triangulation], following is the error message:

---------------------------------------------------------------------
Camera synchronization
On Tuesday 16. July 2024, 10:52:49
---------------------------------------------------------------------

Project directory: C:\Users\Ivan\Desktop\single_trial
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 Pose2Sim.synchronization()

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:307, in synchronization(config)
    304 logging.info("---------------------------------------------------------------------")
    305 logging.info(f"\nProject directory: {project_dir}")
--> 307 synchronize_cams_all(config_dict)
    309 end = time.time()
    310 elapsed = end-start

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\synchronization.py:295, in synchronize_cams_all(config_dict)
    293 except:
    294     raise ValueError(f'No json files found in {pose_dir}. Make sure you run Pose2Sim.poseEstimation() first.')
--> 295 pose_listdirs_names = sort_stringlist_by_last_number(pose_listdirs_names)
    296 json_dirs_names = [k for k in pose_listdirs_names if 'json' in k]
    297 json_dirs = [os.path.join(pose_dir, j_d) for j_d in json_dirs_names] # list of json directories in pose_dir

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:355, in sort_stringlist_by_last_number(string_list)
    352 def sort_by_last_number(s):
    353     return int(re.findall(r'\d+', s)[-1])
--> 355 return sorted(string_list, key=sort_by_last_number)

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:353, in sort_stringlist_by_last_number.<locals>.sort_by_last_number(s)
    352 def sort_by_last_number(s):
--> 353     return int(re.findall(r'\d+', s)[-1])

IndexError: list index out of range

In [5]: Pose2Sim.personAssociation()

---------------------------------------------------------------------
Associating persons for single_trial, for all frames.
On Tuesday 16. July 2024, 10:54:12
---------------------------------------------------------------------

Project directory: C:\Users\Ivan\Desktop\single_trial
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 Pose2Sim.personAssociation()

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:354, in personAssociation(config)
    351 logging.info("---------------------------------------------------------------------")
    352 logging.info(f"\nProject directory: {project_dir}")
--> 354 track_2d_all(config_dict)
    356 end = time.time()
    357 elapsed = end-start

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\personAssociation.py:672, in track_2d_all(config_dict)
    670 # 2d-pose files selection
    671 pose_listdirs_names = next(os.walk(pose_dir))[1]
--> 672 pose_listdirs_names = sort_stringlist_by_last_number(pose_listdirs_names)
    673 json_dirs_names = [k for k in pose_listdirs_names if 'json' in k]
    674 try:

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:355, in sort_stringlist_by_last_number(string_list)
    352 def sort_by_last_number(s):
    353     return int(re.findall(r'\d+', s)[-1])
--> 355 return sorted(string_list, key=sort_by_last_number)

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:353, in sort_stringlist_by_last_number.<locals>.sort_by_last_number(s)
    352 def sort_by_last_number(s):
--> 353     return int(re.findall(r'\d+', s)[-1])

IndexError: list index out of range

In [6]: Pose2Sim.triangulation()

---------------------------------------------------------------------
Triangulation of 2D points for single_trial, for all frames.
On Tuesday 16. July 2024, 10:54:20
---------------------------------------------------------------------

Project directory: C:\Users\Ivan\Desktop\single_trial
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 Pose2Sim.triangulation()

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:400, in triangulation(config)
    397 logging.info("---------------------------------------------------------------------")
    398 logging.info(f"\nProject directory: {project_dir}")
--> 400 triangulate_all(config_dict)
    402 end = time.time()
    403 elapsed = end-start

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\triangulation.py:782, in triangulate_all(config_dict)
    780 # 2d-pose files selection
    781 pose_listdirs_names = next(os.walk(pose_dir))[1]
--> 782 pose_listdirs_names = sort_stringlist_by_last_number(pose_listdirs_names)
    783 json_dirs_names = [k for k in pose_listdirs_names if 'json' in k]
    784 n_cams = len(json_dirs_names)

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:355, in sort_stringlist_by_last_number(string_list)
    352 def sort_by_last_number(s):
    353     return int(re.findall(r'\d+', s)[-1])
--> 355 return sorted(string_list, key=sort_by_last_number)

File ~\miniconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\common.py:353, in sort_stringlist_by_last_number.<locals>.sort_by_last_number(s)
    352 def sort_by_last_number(s):
--> 353     return int(re.findall(r'\d+', s)[-1])

IndexError: list index out of range

It seems to me the three error messages are identical, though I cannot figure out which part I did goes wrong.

To answer your thought:

1. Yes i did reinstall a new conda environment to ensure gpu accleration is available, the python version is 3.9.19
2. I ran two trials with different calibration process(the video folder is identical), both were using checkerboard, one is using Pose2Sim calculation(I manually clicked corners on checkerboard hoping for a more precise result) and one is converted from easyMocap. However both trials gave back the same error message above.
3. I curretly switch to folder structure to a single trial.

Thank you for your reply!:)

[davidpagnon]

Hi, Looking at your error, I can see two reasons for it to crash:

• you do not run pose2sim from the right folder (I doubt it)
• your pose folders are empty, or do not end with '_json'
• your json files do not contain a number

But I could be missing something. Can you tell me the structure of your directory?

You can use this function for example:

def list_files(startpath):
    for root, dirs, files in os.walk(startpath):
        level = root.replace(startpath, '').count(os.sep)
        indent = ' ' * 4 * (level)
        print('{}{}/'.format(indent, os.path.basename(root)))
        subindent = ' ' * 4 * (level + 1)
        for f in files:
            print('{}{}'.format(subindent, f))

list_files(pose_dir)

[davidpagnon]

Sidenote: extrinsic calibration based on a checkerboard is generally less accurate, because you calibrate on a single plane and on a small surface. So even if your reprojection error is low on the checkerboard points, you could obtain higher reprojection errors on triangulation. However, if you do need to use a checkerboard, it is usually best to let the corners be detected automatically (unless it clearly looks like they are wrongly detected). You always get a few pixels off by clicking, while the automatic detection has a subpixel accuracy.

[peterlololsss]

Hi, sorry for the late reply

Followign is the folder structure,(Very long due to json files.)

C:\Users\Ivan\miniconda3\python.exe D:\PyCharm\PyCharmProjects\PoseEstimation\List_files.py 
single_trial/
    Config.toml
    logs.txt
    Calibration/
        Calib_board.toml
        Object_points.trc
        extrinsics/
            logi_1/
                extrinsic_image_1_0000.png
            logi_2/
                extrinsic_image_1_0000.png
            logi_3/
                extrinsic_image_1_0000.png
            Osmo/
                extrinsic_image_1_0000.png
        intrinsics/
            logi_1/
                intrinsic_image_0000.jpg
to
intrinsic_image_0053.jpg
            logi_2/
                intrinsic_image_0000.jpg
to
intrinsic_image_0053.jpg
            logi_3/
                intrinsic_image_0000.jpg
to
intrinsic_image_0053.jpg
            Osmo/
                intrinsic_image_0000.jpg
to
intrinsic_image_0053.jpg
    pose/
        logi_1_pose.mp4
        logi_2_pose.mp4
        logi_3_pose.mp4
        osmo_pose.mp4
        logi_1_json/
            logi_1_000000.json
            to
            logi_1_000348.json
        logi_2_json/
            logi_2_000000.json
            to
            logi_2_000348.json
        logi_3_json/
            logi_3_000000.json
            to
            logi_3_000348.json
        osmo_json/
            osmo_000000.json 
            to
            osmo_000348.json
    videos/
        logi_1.mp4
        logi_2.mp4
        logi_3.mp4
        osmo.mp4

Process finished with exit code 0

My confusions are:

1. If I do not run pose2sim from the right folder, then poseEstimaiton() would not work in that case, Yet it did results in (I assume?) desired json files from above log.
2. From the looks of it the json files are... ok? And when i looked into one of the json files, the pose_keypoints_2d has arrays with stats
3. I would like to recalibrate using scene measurement to try again, however I do not fully grasp the meaning of the method. For example, if I put a table on the center of capture videos areas, do I just choosing points based on my own spacial awareness?

[davidpagnon]

1. Yes, that's why I doubted it was the problem
2. I cannot see any issue in the file structure. Would you mind sending me your folder at dp2032@bath.ac.uk? That will help me find what's wrong
3. A table is just a simple 3D object that can be found anywhere. The idea is that it is best to find something either larger than a checkerboard, or 3D for a better calibration. What you need to do is decide where the origin of your frame of reference is, and then measure the coordinates of some points (with a tape measurer for example)

[davidpagnon]

Thank you for your data!

1. I finally pinned your problem: the Osmo camera does not have a number, so my sorting method fails. I'll push the fix in the next release, but in the meantime you can edit sort_stringlist_by_last_number in Pose2Sim/common.py (do pip show pose2sim to check where the package is located), and replace it by:

   def sort_stringlist_by_last_number(string_list):
   '''
   Sort a list of strings based on the last number in the string.
   Works if other numbers in the string, if strings after number. Ignores alphabetical order.
   
   Example: ['json1', 'zero', 'js4on2.b', 'aaaa', 'eypoints_0000003.json', 'ajson0', 'json10']
   gives: ['ajson0', 'json1', 'js4on2.b', 'eypoints_0000003.json', 'json10', 'aaaa', 'zero']
   '''
   
   def sort_by_last_number(s):
       numbers = re.findall(r'\d+', s)
       if numbers:
           return (False, int(numbers[-1]))
       else:
           return (True, s)
   
   return sorted(string_list, key=sort_by_last_number)

   Or you can just rename your camera so that it has a number in it.

2. Then, triangulation will not work, because your calibration is not valid (all rotations and translations are set to zero).

   • To calculate them with a checkerboard, edit your Config.toml file and set:

     calculate_extrinsics = true
     extrinsics_method = 'board'

   • I doubt you'll get correct results, since the resolution of the images in intrinsic and extrinsic folders is not the same as the ones in your videos. I would suggest taking videos of the checkerboard and of the scene rather than pictures, as even with the same resolution the video is often cropped (for stabilization)

3. Now the questions you asked per email:

   Additionally I have a questions regarding to scene measurement: In my understanding, the measure steps are:(Still takes a table as an example) 1.I choose the origin point 2.Based on the reference of 8 points in the graph from Pose2Sim.calculation(), I found the respective 8 points in the space around the table(Or the corners/edges of the table) 3.Using a measurer to measure the distance from the 8 points to the origin I chose, in (X,Y,Z) axies coordiates 4.Type the Axies coordiates into the config file. 5.Run Pose2Sim calibration Is my understanding correct?

   Not exactly: you need to put your table in the scene, and to measure its coordinates. For example, for a table that is 2 m long, 1 m deep, and 80 cm heigh, it could be:

   object_coords_3d =   [[0.0,  0.0,  0.0], 
                          [2.0 , 0.0,  0.0], 
                          [0.0, 1.0,  0.0], 
                          [2.0, 1.0 ,  0.0], 
                          [0.0,  0.0,  0.8], 
                          [2.0 , 0.0,  0.8], 
                          [0.0, 1.0,  0.8], 
                          [2.0, 1.0 ,  0.8]]

   Then you replace the coordinates in Config.toml by these. You need to set extrinsics_method = 'scene', and then you just follow the instructions and click the corresponding points in the image! Try it and tell me if anything is still unclear.

[peterlololsss]

Thank you for your help! Synchronizations work now after modifying the function:)

And as you said, triangulation doesn't work for the calibration I sent you. But before I recalibrate using scene measurement, I also ran all the stages for the one other trial I calibrated using easymocap, triangulation works however the opensim's run button (After I loaded respective files)grayed out for the .trc file unless I MarkerAugmented it, yet the resulting motion is still horrendous.

I blamed this to my calibration, I would try again with new scene measurement calibration in coming days, I would update to you as soon as I finish the work.

Again, thank you for your help!

[peterlololsss]

Hi, these days I ran some tests and am facing new challenges:(((

the Pose2Sim process is completed however the resulting opensim model animation is twitching and glitching here and there,

Regarding to this I have following concerns:

1. When open a model in opensim, is it the best to open the model with respective pose estimation model?( In my case is COCO_133)

2. The next problem is when loading the .trc file from Pose2Sim, opensim won't let me scale the model with it as shown in the image(No matter was it marker agumented/filtered) Yet if I change the model to body25_scaled or LSTM , it is runnable, but after the ik loaded and run the joint angle is all wrong and the skeleton is twitching. Screenshot 2024-07-23 214159

3. Sometimes opensim would only run 50frames and that's it, not a single frame more...

4. I also tried pose estimating with HALPE_26 along with HALPE_26 model in Opensim_Setup, it is 'runanble' but glitchy and twiching, also (refering to point 3) 50 frames.

5. I am kinda at loss of the direction to look for where the problem is. I recalibrated with scene measurements, and the error rate was around 10mm ---(note: when typing to this point, i realize when calibrating extrinsic parameter I chose 8 reference points which is fewer than recommended 10 points, I would recalibrate tomorrow to see if anything improved)

Besides point 5, is there any advice on where i should look into:)

Triangulation Info : --> Mean reprojection error for all points on all frames is 8.9 px, which roughly corresponds to 13.5 mm. Cameras were excluded if likelihood was below 0.3 and if the reprojection error was above 15 px. Gaps were interpolated with cubic method if smaller than 10 frames. Larger gaps were filled with the last valid value. In average, 0.69 cameras had to be excluded to reach these thresholds. Camera logi_2 was excluded 55% of the time, Camera logi_1: 8%, and Camera logi_3: 6%.

note: I delete the additional osmo camera since its inclusion would result logi_1 and logi_2 to be excluded completely, I may need to readjust its position, I wonder that could be that the osmo's ultrawide view has already make poseEstimation have many errors, which negatively affects the performance of Triangulation .

Thank you for your help along the way! :))))

[davidpagnon]

Hi, I'm not exactly sure what the problem is.

1. Yes, the model you use for pose estimation should be the same as the one you scale in OpenSim, unless you do marker augmentation (in that case it needs to be LSTM).
2. Can you check if there is anything in red in the other tabs? Otherwise, you can try to reload your TRC file, sometimes it works strangely.
3. I've never had this problem of IK running for only 50 frames... Did you set your time range to the whole trial? Does it just stop or is it crashing? If so, can you give me the error message?
4. Are you sure of the quality of your triangulation? Calibration and Synchronization are the usual sources of error. Did you try to look at your triangulation results? You can do it either by using the Blender add-on, or in OpenSim with File -> Preview experimental data.
5. 10 mm and 8 points should work if the points are spaced out enough. Best calibrations are obtained when the points take as much space as possible in the scene, and it is even better if they are 3D instead of on a single plane.

After checking your triangulation info, it does not look bad but you should still try to visualize them. Logi_2 is excluded more than half of the time though, you may want to check its calibration. If you had to delete the additional Osmo, it may be that its calibration was not great. Can you see large distortions of straight lines? I started implementing distortion correction but did not find time to complete it. If you really need it, you can use openCV to undistort the video, and then perform calibration and triangulation again (unless there is an option to film in "Linear" mode like on GoPros).

[peterlololsss]

Hey there,

• After the recalibrated, (also adjusted the camera position to 30 degree angels), the twiching is vastly imporved, I would not say it is perfect but neither was my calibration, so hooray? hooray!

• coco133 opensim model still cannot load the trc file however LSTM model works, I would try other models(like HALPE_26) later to see if that is some compatibility issue from coco133 maybe?

• About the 50 frames, there is a time range setting in opensim which i completely ignored before, it automatically capped the frame. Once changed the issue is gone.

The triangulation info has also improved: Mean reprojection error for all points on all frames is 8.1 px, which roughly corresponds to 15.3 mm. Cameras were excluded if likelihood was below 0.3 and if the reprojection error was above 15 px. Gaps were interpolated with cubic method if smaller than 10 frames. Larger gaps were filled with the last valid value. In average, 0.24 cameras had to be excluded to reach these thresholds. Camera logi_2 was excluded 12% of the time, Camera logi_4: 9%, Camera logi_1: 2%, and Camera logi_3: 1%.

It is a real surprise after trying to figure out those issues during these days, it cannot done without your help so here's my gratitude :))

[davidpagnon]

I'm glad it works now! I'll have to have a look if there is an issue with the Coco_133 model.

[davidpagnon]

Can you tell me what failed with Body_133? I cannot reproduce the issue.

[peterlololsss]

Hi, It seems there are markers not found shown in the screenshots between .trc files and both ik and scale setups.(Red areas in the screenshots).

Once I unclicked those problem markers or change scaling factor manually, it is runnable then.

[davidpagnon]

Okay, thanks! I'll release a new version soon, but I think the error should be fixed if you change lines 375 and 376 in skeletons.py:

            Node("Reye", id=2),
            Node("Leye", id=1),

Can you tell me if it works?

[peterlololsss]

Yep that worked. But noted it is REye with the capital E or it wont. thank you for your help:)

[davidpagnon]

Thank you for the feedback!