search

perfanalytics / pose2sim

Markerless kinematics with any cameras — From 2D Pose estimation to 3D OpenSim motion
https://perfanalytics.github.io/pose2sim/
BSD 3-Clause "New" or "Revised" License
260 stars 49 forks source link

extrinsics calibration - list index out of range #101

Closed EmanuelWicki closed 6 months ago

EmanuelWicki commented 6 months ago

Hi Daniel, I am encountering a problem while trying to calculate my extrinsics using a scene. I have 11 points in my scene (all on the floor -> 2D) and after clicking on all the points in my scene I'm getting the following error. Do I have to change the index range? 

Retrieving intrinsic parameters from file. Set "overwrite_intrinsics" to true in Config.toml to recalculate them.        

Calculating extrinsic parameters...

Calculating extrinsic parameters...

Camera ext_cam01_img:
Installed qt5 event loop hook.
BOARD EXECUTED !!!!!!!
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 Pose2Sim.calibration()

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:208, in calibration(config)
    205 logging.info(f"\nCalibration directory: {calib_dir}")
    206 start = time.time()
--> 208 calibrate_cams_all(config_dict)
    210 end = time.time()
    211 logging.info(f'Calibration took {end-start:.2f} s.')

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\calibration.py:1351, in calibrate_cams_all(config)
   1348 calib_fun = calib_mapping[calib_full_type]
   1350 # Calibrate
-> 1351 ret, C, S, D, K, R, T = calib_fun(*args_calib_fun)
   1353 # Write calibration file
   1354 toml_write(calib_output_path, C, S, D, K, R, T)

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\calibration.py:532, in calib_calc_fun(calib_dir, intrinsics_config_dict, extrinsics_config_dict)
    530 if calculate_extrinsics:
    531     logging.info(f'\nCalculating extrinsic parameters...')
--> 532     ret, C, S, D, K, R, T = calibrate_extrinsics(calib_dir, extrinsics_config_dict, C, S, K, D)
    533 else:
    534     logging.info(f'\nExtrinsic parameters won\'t be calculated. Set "calculate_extrinsics" to true in Config.toml to calculate them.')

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\calibration.py:711, in calibrate_extrinsics(calib_dir, extrinsics_config_dict, C, S, K, D)
    709 # Calculate extrinsics
    710 print('BOARD EXECUTED !!!!!!!')
--> 711 mtx, dist = np.array(K[i]), np.array(D[i])
    712 _, r, t = cv2.solvePnP(np.array(object_coords_3d), imgp, mtx, dist) # changed objp to object_coords_3d
    713 r, t = r.flatten(), t.flatten()

IndexError: list index out of range

Also the calibration part of my calib.toml file is attached here:

[calibration]
calibration_type = 'calculate' # 'convert' or 'calculate'

   [calibration.convert]
   convert_from = 'qualisys' # 'qualisys', 'optitrack', vicon', 'opencap', 'easymocap', 'biocv', 'anipose', or 'freemocap'
      [calibration.convert.qualisys]
      binning_factor = 1 # Usually 1, except when filming in 540p where it usually is 2
      [calibration.convert.optitrack]  # See readme for instructions
      [calibration.convert.vicon]      # No parameter needed
      [calibration.convert.opencap]    # No parameter needed
      [calibration.convert.easymocap]  # No parameter needed
      [calibration.convert.biocv]      # No parameter needed
      [calibration.convert.anipose]    # No parameter needed
      [calibration.convert.freemocap]  # No parameter needed

   [calibration.calculate] 
      # Camera properties, theoretically need to be calculated only once in a camera lifetime
      [calibration.calculate.intrinsics]
      overwrite_intrinsics = false # overwrite (or not) if they have already been calculated?
      show_detection_intrinsics = true # true or false (lowercase)
      intrinsics_extension = 'avi' # any video or image extension
      extract_every_N_sec = 1 # if video, extract frames every N seconds (can be <1 )
      intrinsics_corners_nb = [6,8] 
      intrinsics_square_size = 40 # mm

      # Camera placements, need to be done before every session
      [calibration.calculate.extrinsics]
      calculate_extrinsics = true # true or false (lowercase) 
      extrinsics_method = 'scene' # 'board', 'scene', 'keypoints'
      # 'board' should be large enough to be detected when laid on the floor. Not recommended.
      # 'scene' involves manually clicking any point of know coordinates on scene. Usually more accurate if points are spread out.
      # 'keypoints' uses automatic pose estimation of a person freely walking and waving arms in the scene. Slighlty less accurate, requires synchronized cameras.
      moving_cameras = false # Not implemented yet

         [calibration.calculate.extrinsics.board]
         show_reprojection_error = true # true or false (lowercase)
         extrinsics_extension = 'avi' # any video or image extension
         extrinsics_corners_nb = [3,5] # [H,W] rather than [w,h]
         extrinsics_square_size = 65 # mm # [h,w] if square is actually a rectangle

         [calibration.calculate.extrinsics.scene]
         show_reprojection_error = true # true or false (lowercase)
         extrinsics_extension = 'avi' # any video or image extension
         # list of 3D coordinates to be manually labelled on images. Can also be a 2 dimensional plane. 
         # in m -> unlike for intrinsics, NOT in mm!
         object_coords_3d =   [[0.0,  0.0,  0.0], 
                              [0.603 , 0.0,  0.0], 
                              [1.206, 0.0,  0.0], 
                              [0.0, 0.404 ,  0.0], 
                              [0.603,  0.404,  0.0], 
                              [1.206, 0.404,  0.0], 
                              [-0.182, 0.604,  0.0], 
                              [-0.105, 0.604,  0.0],
                              [-0.182, 1.206,  0.0],
                              [-0.105, 1.206,  0.0],
                              [1.942, 1.206,  0.0]]

         [calibration.calculate.extrinsics.keypoints]
         # Coming soon!"
davidpagnon commented 6 months ago

Hi,

It's telling you that the length of K is lesser than i. K comes from the previous intrinsic calibration you have done. Can you check your calibration file and see if you have as many cameras as the number in your extrinsics folder?

EmanuelWicki commented 6 months ago

my calib_board.toml "[int_cam01_img] name = "int_cam01_img" size = [ 1920.0, 1088.0] matrix = [ [ 1665.8086662960393, 0.0, 959.5], [ 0.0, 1676.4957228942644, 543.5], [ 0.0, 0.0, 1.0]] distortions = [ -0.04141344589689808, 0.07623007428998166, 0.002190791974704825, 0.0025334756418212917] rotation = [ -0.028717891075011447, -2.1733186665096875, -2.259349446007465] translation = [ -1.1092073314704791, 0.8169849411116984, 2.947136342466439] fisheye = false

[int_cam02_img] name = "int_cam02_img" size = [ 1920.0, 1088.0] matrix = [ [ 1667.3666363291732, 0.0, 959.5], [ 0.0, 1675.9873681252775, 543.5], [ 0.0, 0.0, 1.0]] distortions = [ -0.045115671761216065, 0.0881454168156573, 0.0047135447214286495, -0.0006271002821147498] rotation = [ -1.0960041663577, 1.1158102553797193, 1.2824616425271775] translation = [ 0.11288934582699561, 0.6848357402632919, 3.5472627091515085] fisheye = false

[metadata] adjusted = false error = 0.0 " I'm encountering an issue with how the system accesses the int_cam01_img folder during the extrinsic calibration, as noted in the message below. To prevent this, I deleted the folders within the intrinsic directory so they couldn't be accessed during the external calibration, and this process proceeded without errors. Could this approach have caused the previous error ?

Moreover, whenever there are files in the int_cam01_img folder, they are automatically used for extrinsic calibration. Since I used a checkerboard for intrinsic calibration, I believe its not necessary to identify checkerboard corners during the extrinsic calibration, right.

"Retrieving intrinsic parameters from file. Set "overwrite_intrinsics" to true in Config.toml to recalculate them.

Calculating extrinsic parameters...

Camera int_cam01_img: Extracting frames... Test 0002_Miqus_15_22269_00000.png: Corners found. Installed qt5 event loop hook. Test 0002_Miqus_15_22269_00050.png: Corners found. Test 0002_Miqus_15_22269_00100.png: Corners found. Test 0002_Miqus_15_22269_00150.png: Corners found."

davidpagnon commented 6 months ago

That may be the cause, try to retrieve the intrinsic camera folders, and to set overwrite_intrinsics to false instead.

Or are you telling me that even with this parameter, it is trying to recalculate the intrinsic parameters anyway?

EmanuelWicki commented 6 months ago

I have now done the whole process again: intrinsic and extrinsic calibration - that way it worked fine but even now i had to calibrate intrinsics twice as can be seen below. Before I had overwrite_intrinsics to false but it still tried to access the images in the int_cam01_img.

"Calculating intrinsic parameters...

Camera int_cam01_img: Test 0002_Miqus_15_22269_00000.png: Corners found. Test 0002_Miqus_15_22269_00050.png: Corners found. Test 0002_Miqus_15_22269_00100.png: Corners found. Test 0002_Miqus_15_22269_00150.png: Corners found. Test 0002_Miqus_15_22269_00200.png: Corners found. Test 0002_Miqus_15_22269_00250.png: Corners found. Test 0002_Miqus_15_22269_00300.png: Corners found. Test 0002_Miqus_15_22269_00350.png: Corners found. Test 0002_Miqus_15_22269_00400.png: Corners found. Test 0002_Miqus_15_22269_00450.png: Corners found. Test 0002_Miqus_15_22269_00500.png: Corners found. Test 0002_Miqus_15_22269_00550.png: Corners found. Test 0002_Miqus_15_22269_00600.png: Corners found. Test 0002_Miqus_15_22269_00650.png: Corners found. Test 0002_Miqus_15_22269_00700.png: Corners found. Test 0002_Miqus_15_22269_00750.png: Corners found. Test 0002_Miqus_15_22269_00800.png: Corners found. Test 0002_Miqus_15_22269_00850.png: Corners found. Test 0002_Miqus_15_22269_00900.png: Corners found. Test 0002_Miqus_15_22269_00950.png: Corners found. Test 0002_Miqus_15_22269_01000.png: Corners found. Test 0002_Miqus_15_22269_01050.png: Corners found. Test 0002_Miqus_15_22269_01100.png: Corners found. Test 0002_Miqus_15_22269_01150.png: Corners found. Test 0002_Miqus_15_22269_01200.png: Corners found. Test 0002_Miqus_15_22269_01250.png: Corners found. Test 0002_Miqus_15_22269_01300.png: Corners found. Test 0002_Miqus_15_22269_01350.png: Corners found. Test 0002_Miqus_15_22269_01400.png: Corners found. Test 0002_Miqus_15_22269_01450.png: Corners found. Intrinsics error: 0.211 px for each cameras.

Camera int_cam02_img: Test 0004_Miqus_16_22270_00000.png: Corners found. Test 0004_Miqus_16_22270_00100.png: Corners found. Test 0004_Miqus_16_22270_00200.png: Corners found. Test 0004_Miqus_16_22270_00300.png: Corners found. Test 0004_Miqus_16_22270_00400.png: Corners found. Test 0004_Miqus_16_22270_00500.png: Corners found. Test 0004_Miqus_16_22270_00600.png: Corners found. Test 0004_Miqus_16_22270_00700.png: Corners found. Test 0004_Miqus_16_22270_00800.png: Corners found. Test 0004_Miqus_16_22270_00900.png: Corners found. Test 0004_Miqus_16_22270_01000.png: Corners found. Test 0004_Miqus_16_22270_01100.png: Corners found. Test 0004_Miqus_16_22270_01200.png: Corners found. Test 0004_Miqus_16_22270_01300.png: Corners found. Test 0004_Miqus_16_22270_01400.png: Corners found. Corners were detected only on 8 images for camera int_cam02_img. Calibration of intrinsic parameters may not be accurate with fewer than 10 good images of the board. Intrinsics error: 0.248 px for each cameras.

Calculating extrinsic parameters...

Camera int_cam01_img: Test 0002_Miqus_15_22269_00000.png: Corners found. Test 0002_Miqus_15_22269_00050.png: Corners found. Test 0002_Miqus_15_22269_00100.png: Corners found. Test 0002_Miqus_15_22269_00150.png: Corners found. Test 0002_Miqus_15_22269_00200.png: Corners found. Test 0002_Miqus_15_22269_00250.png: Corners found. Test 0002_Miqus_15_22269_00300.png: Corners found. Test 0002_Miqus_15_22269_00350.png: Corners found. Test 0002_Miqus_15_22269_00400.png: Corners found. Test 0002_Miqus_15_22269_00450.png: Corners found. Test 0002_Miqus_15_22269_00500.png: Corners found. Test 0002_Miqus_15_22269_00550.png: Corners found. Test 0002_Miqus_15_22269_00600.png: Corners found. Test 0002_Miqus_15_22269_00650.png: Corners found. Test 0002_Miqus_15_22269_00700.png: Corners found. Test 0002_Miqus_15_22269_00750.png: Corners found. Test 0002_Miqus_15_22269_00800.png: Corners found. Test 0002_Miqus_15_22269_00850.png: Corners found. Test 0002_Miqus_15_22269_00900.png: Corners found. Test 0002_Miqus_15_22269_00950.png: Corners found. Test 0002_Miqus_15_22269_01000.png: Corners found. Test 0002_Miqus_15_22269_01050.png: Corners found. Test 0002_Miqus_15_22269_01100.png: Corners found. Test 0002_Miqus_15_22269_01150.png: Corners found. Test 0002_Miqus_15_22269_01200.png: Corners found. Test 0002_Miqus_15_22269_01250.png: Corners found. Test 0002_Miqus_15_22269_01300.png: Corners found. Test 0002_Miqus_15_22269_01350.png: Corners found. Test 0002_Miqus_15_22269_01400.png: Corners found. Test 0002_Miqus_15_22269_01450.png: Corners found. Intrinsics error: 0.211 px for each cameras.

Camera int_cam02_img: Test 0004_Miqus_16_22270_00000.png: Corners found. Test 0004_Miqus_16_22270_00100.png: Corners found. Test 0004_Miqus_16_22270_00200.png: Corners found. Test 0004_Miqus_16_22270_00300.png: Corners found. Test 0004_Miqus_16_22270_00400.png: Corners found. Test 0004_Miqus_16_22270_00500.png: Corners found. Test 0004_Miqus_16_22270_00600.png: Corners found. Test 0004_Miqus_16_22270_00700.png: Corners found. Test 0004_Miqus_16_22270_00800.png: Corners found. Test 0004_Miqus_16_22270_00900.png: Corners found. Test 0004_Miqus_16_22270_01000.png: Corners found. Test 0004_Miqus_16_22270_01100.png: Corners found. Test 0004_Miqus_16_22270_01200.png: Corners found. Test 0004_Miqus_16_22270_01300.png: Corners found. Test 0004_Miqus_16_22270_01400.png: Corners found. Corners were detected only on 8 images for camera int_cam02_img. Calibration of intrinsic parameters may not be accurate with fewer than 10 good images of the board. Intrinsics error: 0.248 px for each cameras.

Calculating extrinsic parameters...

Camera ext_cam01_img: BOARD EXECUTED !!!!!!!

Camera ext_cam02_img: BOARD EXECUTED !!!!!!!

--> Residual (RMS) calibration errors for each camera are respectively [5.594, 3.867] px, which corresponds to [10.682, 8.547] mm. "

davidpagnon commented 6 months ago

That's strange, did you change the code? Can you try on the demo data to check if it does it there too?

EmanuelWicki commented 6 months ago

I dont remember chaning the code and yes also with the demo it happens... I will clone the repo again and try if it still happens

davidpagnon commented 6 months ago

Okay, keep me updated, and if it still happens I will dive a bit more into the code!

(this may also help narrowing down the problem: you installed it via cloning the repo rather than with pip install pose2sim?)

EmanuelWicki commented 6 months ago

I cloned the repo (also before) and tried to run the demo but I cant calculate personAssociation for the Batch Sesion file, the BODY25b model is selected and neck is tracked...

"--------------------------------------------------------------------- Associating persons for S00_P00_T00_StaticTrial, for all frames. On Friday 26. April 2024, 10:24:50

Project directory: C:\Users...\S00_Demo_BatchSession\S00_P00_SingleParticipant\S00_P00_T00_StaticTrial

Single-person analysis selected.

IndexError Traceback (most recent call last) Cell In[4], line 1 ----> 1 Pose2Sim.personAssociation()

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:348, in personAssociation(config)
345 logging.info("---------------------------------------------------------------------") 346 logging.info(f"\nProject directory: {project_dir}") --> 348 track_2d_all(config_dict) 350 end = time.time() 351 elapsed = end-start

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\personAssociation.py:436, in track_2d_all(config) 434 except: 435 raise NameError('Model not found in skeletons.py nor in Config.toml') --> 436 tracked_keypointid = [node.id for , _, node in RenderTree(model) if node.name==tracked_keypoint][0] 438 # 2d-pose files selection 439 pose_listdirs_names = next(os.walk(pose_dir))[1]

IndexError: list index out of range"

For the S01_Demo_Single_Trial i tried as well but for Pose2Sim.calibration() i got the error "Cell In[3], line 1 ----> 1 Pose2Sim.calibration()

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:190, in calibration(config) 180 ''' 181 Cameras calibration from checkerboards or from qualisys files. 182 (...) 185 or the function can be called without an argument, in which case it the config directory is the current one.
186 ''' 188 from Pose2Sim.calibration import calibrate_cams_all --> 190 level, config_dicts = read_config_files(config) 191 config_dict = config_dicts[0] 192 session_dir = os.path.realpath([os.getcwd() if level==3 else os.path.join(os.getcwd(), '..') if level==2 else os.path.join(os.getcwd(), '..', '..')][0])

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:129, in read_config_files(config)
127 # Trial level 128 if level == 1: --> 129 session_config_dict = toml.load(os.path.join(config_dir, '..','..','Config.toml')) 130 participant_config_dict = toml.load(os.path.join(config_dir, '..','Config.toml')) 131 trial_config_dict = toml.load(os.path.join(config_dir, 'Config.toml'))

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\toml\decoder.py:133, in load(f, _dict, decoder) 114 """Parses named file or files as toml and returns a dictionary 115 116 Args: (...) 129 (Python 2 / Python 3) file paths is passed 130 """ 132 if _ispath(f): --> 133 with io.open(_getpath(f), encoding='utf-8') as ffile: 134 return loads(ffile.read(), _dict, decoder) 135 elif isinstance(f, list):

FileNotFoundError: [Errno 2] No such file or directory: '.\..\..\Config.toml'"

davidpagnon commented 6 months ago

Hi again, You haven't told me how calibration went, did it solve it? I suppose you ran personAssociation() directly from S00_P00_T00_StaticTrial? I may have found a bug that I introduced in the last release, can you tell me if it works now?

EmanuelWicki commented 6 months ago

Hi David, yes i ran personAssiciation() directly from S00_P00_T00_StaticTrial but it still didnt work just now. I still got the following error (For the Demo files):

"Project directory: C:\Users...\S00_Demo_BatchSession\S00_P00_SingleParticipant\S00_P00_T00_StaticTrial

Single-person analysis selected.

IndexError Traceback (most recent call last) Cell In[4], line 1 ----> 1 Pose2Sim.personAssociation()

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py:348, in personAssociation(config)
345 logging.info("---------------------------------------------------------------------") 346 logging.info(f"\nProject directory: {project_dir}") --> 348 track_2d_all(config_dict) 350 end = time.time() 351 elapsed = end-start

File C:\ProgramData\anaconda3\envs\Pose2Sim\lib\site-packages\Pose2Sim\personAssociation.py:436, in track_2d_all(config) 434 except: 435 raise NameError('Model not found in skeletons.py nor in Config.toml') --> 436 tracked_keypointid = [node.id for , _, node in RenderTree(model) if node.name==tracked_keypoint][0] 438 # 2d-pose files selection 439 pose_listdirs_names = next(os.walk(pose_dir))[1]

IndexError: list index out of range"

With my own data it works. The only thing is that I still have negative z coordinates even though I use scene now and set the coordinates of the points in a RH coordinate system...

davidpagnon commented 6 months ago

Hi, it is strange that it does not work on the demo data, since I run automatic tests after each upload which did not report any error, and to be on the safe side I tried it also by myself. Maybe you could manually totally delete your pose2sim folder in your environnent (type pip show pose2sim to find its location, as well as the demo folders??

Anyway, I'm glad it works on your demo data! If the coordinates are still upside down, you just need to label the extrinsics the other way around. For example If you go left and then up, try left and then down.

EmanuelWicki commented 6 months ago

thank you very much, this resolved the issue !