calibration: corner detection doesn't seem right

[delaroob]

SLEAP

Hi everyone!

I am trying to triangulate some SLEAP pose data of a fish using sleap-anipose, but something seems off about the calibration.

(Hopefully,) I followed the instructions of the user guide:

1. using 2 cameras, took calibration images (charuco board, previously generated in SLEAP, kept the .toml file and printed the .jpg) boards_toml_file_content.txt
2. took videos
3. trained SLEAP model
4. selected a video and its pair from the other camera view (the footages were syncronyzed) and analyzed them with the same SLEAP model with great results on both.
5. Using the calibration images taken from the same session, I ran calibration, but when visualizing the predictions, the model didn't find a single corner of the charuco board (on any of the 25-25 frames, which I know that it isn’t a lot but didnt even get far enough to see if it fails because of that:D), even though there was no occulsion, the fields could be seen well.

the mentioned calibration I ran:

slap-calibrate --session '/content/drive/MyDrive/sleap/session_0' --board '/content/drive/MyDrive/sleap/session_0/calibration.toml'  --calib_fname '/content/drive/MyDrive/sleap/session_0/cam_calibrated.toml' --metadata_fname '/content/drive/MyDrive/sleap/session_0/cam_calibrated.h5' --histogram_path '/content/drive/MyDrive/sleap/session_0/histogram.png' --reproj_path '/content/drive/MyDrive/sleap/session_0'

a few examples of the calibration outputs:

Am I missing something? Thanks in advance for any help!

PS: 1) Cameras were placed perpendicular to eachother. Im not sure if that will be a problem later or if that itself is the issue now. 2) Im not sure if calibration could be done with a chessboard but I have some calibration footage with chessboards from the same session if that helps. 3) totally unrelated but I am so very grateful that labels from DLC can be imported, saved me a ton of work, thank you so much!!

[ericleonardis]

Hey, sorry for missing this, our lead developer for this project recently went off to grad school, but now I am helping with maintenance! Can you send your synchronized video to my email eleonardis@salk.edu to see if I can replicate your error and will troubleshoot the OpenCV detection pipeline. In general, I don't think 25 frames will be sufficient for the calibration. Also I would recommend 3 or more cameras to get really reliable 3D estimation. A stereo camera setup is subject to more issues with projective ambiguity. That being said... that is likely not your problem because your detections are not working in the first place.

The ChAruco board looks big enough to be detected and the lighting conditions look alright. Can you send a picture of the camera setup itself? I am not quite sure what you mean by perpendicular but that may contribute to calibration issues. The detection happens so early in the pipeline that it may have little to do with the camera placement.

Is your calibration data just the batch of 25 images? What file format are the images in? Have you tried taking a calibration video rather than the images? When you take the calibration video, make sure all parts of the board are visible to both cameras. It would be helpful to know what version of opencv-python, opencv-contrib-python, and sleap-anipose you are using currently.

[delaroob]

Hi! It's all right, thanks for helping me anyway!

Yes, I absolutely understand the issue with using two cameras, I expected some inaccuracies in the data later on, but since the board detection already failed, I never got that far:)

I did have synchronized calibration videos, but 99% of the footage was unusable because one of the cameras didn't capture all parts of the board, the twenty-something image pairs I used were the exceptions (I know that's not a lot so I didn't expect any wonders, it's just that it didn't even detect the board at all). So I thought next time I would use a smaller charuco board (same board as before just printed it out smaller), it seemed easier to manage and less prone to the above issue.

However, this approach didn't work either, but I've just realized the problem might just be right away with the calibration.toml file: I used the same file as for the larger board, but maybe the parameters square_length and marker_length need adjusting because the board was smaller? Anyway, the calibration process didn't run properly; it exited with an error. It didn't generate the expected reprojection error images, only a cam_calibrated.toml file (and not even cam_calibrated.h5 or the histogram).

Also, I have (hopefully) sent you the synced calibration videos and the camera setup of the newer experiment, the earlier one wouldn't do much good on the long run anyway with so little useable calibration footage.

If there's any other file, output that you would need, please let me know, and thank you so much in advance for the help!

Package versions: opencv-contrib-python (4.6.0.66) sleap-anipose (0.1.7) opencv-python (4.5.5.64)

File structure:

.
└── sleap/
    └── session_1/
        ├── camA/
        │   ├── calibration_images/
        │   │   └── session_1-camA-calibration.mp4
        │   ├── jan_male6_camA.mp4
        │   └── jan_male6_camA.mp4.predictions.000_jan_male6_camA.analysis.h5
        ├── camS/
        │   ├── calibration_images/
        │   │   └── session_1-camS-calibration.mp4
        │   ├── jan_male6_camS.mp4
        │   └── jan_male6_camS.mp4.predictions.000_jan_male6_camA.analysis.h5
        └── calibration.toml

Contents of calibration.toml:

board_x = 11
board_y = 8
square_length = 24.0
marker_length = 18.75
marker_bits = 4
dict_size = 1000

This is all i've done in colab:

!apt update && apt install cuda-11-8

from google.colab import drive
drive.mount('/content/drive/')

!pip install sleap-anipose

input: !slap-calibrate --session '/content/drive/MyDrive/sleap/session_1' --board '/content/drive/MyDrive/sleap/session_1/calibration.toml' --calib_fname '/content/drive/MyDrive/sleap/session_1/cam_calibrated.toml' --metadata_fname '/content/drive/MyDrive/sleap/session_1/cam_calibrated.h5' --histogram_path '/content/drive/MyDrive/sleap/session_1/histogram.png' --reproj_path '/content/drive/MyDrive/sleap/session_1' output:

/content/drive/MyDrive/sleap/session_1/camA/calibration_images/session_1-camA-calibration.mp4
100%|█████████████████████████████| 5544/5544 [07:01<00:00, 13.15it/s]
4095 boards detected
/content/drive/MyDrive/sleap/session_1/camS/calibration_images/session_1-camS-calibration.mp4
100%|█████████████████████████████| 5544/5544 [07:03<00:00, 13.09it/s]
5123 boards detected
defaultdict(<class 'int'>, {('camA', 'camS'): 2629, ('camS', 'camA'): 2629})
error:  28.097275802422153
n_samples: 200
{(0, 1): (1000, array([ 5.70204995, 75.76301472]))}
error: 30.22, mu: 15.0, ratio: 0.346
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         1.8049e+05                                    4.91e+04    
       1              2         4.2244e+04      1.38e+05       4.74e+03       4.29e+04    
       2              3         1.7535e+04      2.47e+04       3.05e+03       1.07e+05    
       3              5         1.0953e+04      6.58e+03       1.22e+03       4.28e+04    
       4              7         9.2460e+03      1.71e+03       2.93e+02       2.67e+04    
       5              8         8.6111e+03      6.35e+02       3.82e+02       1.87e+03    
       6             10         8.4554e+03      1.56e+02       1.25e+02       1.75e+03    
       7             11         8.4028e+03      5.26e+01       1.23e+02       2.47e+03    
       8             12         8.3455e+03      5.72e+01       4.28e+02       8.33e+03    
       9             14         8.3144e+03      3.12e+01       2.23e+02       9.98e+02    
      10             15         8.2576e+03      5.67e+01       3.22e+02       1.27e+04    
      11             16         8.2045e+03      5.31e+01       8.44e+02       4.43e+04    
      12             17         8.0802e+03      1.24e+02       4.51e+02       2.38e+04    
      13             19         8.0208e+03      5.95e+01       4.19e+02       1.75e+04    
      14             21         7.9979e+03      2.29e+01       2.12e+02       1.49e+04    
      15             23         7.9878e+03      1.01e+01       8.91e+01       1.93e+04    
      16             25         7.9814e+03      6.44e+00       3.30e+01       1.57e+04    
      17             26         7.9665e+03      1.48e+01       2.42e+01       1.13e+04    
      18             27         7.9567e+03      9.80e+00       6.60e+01       2.78e+03    
      19             28         7.9462e+03      1.05e+01       2.38e+02       5.75e+03    
      20             29         7.9299e+03      1.63e+01       5.74e+02       3.07e+03    
      21             30         7.9151e+03      1.48e+01       1.69e+03       1.05e+04    
      22             31         7.9035e+03      1.16e+01       8.64e+02       2.04e+04    
      23             34         7.8964e+03      7.09e+00       4.41e+01       2.67e+04    
      24             35         7.8775e+03      1.89e+01       2.46e+01       1.02e+04    
      25             36         7.8729e+03      4.61e+00       6.39e+01       2.99e+02    
      26             37         7.8684e+03      4.51e+00       2.44e+02       2.66e+03    
      27             38         7.8623e+03      6.09e+00       7.80e+02       8.39e+02    
      28             39         7.8555e+03      6.86e+00       1.86e+03       1.66e+03    
      29             41         7.8549e+03      5.70e-01       9.43e+02       6.06e+02    
      30             43         7.8516e+03      3.32e+00       2.19e+01       3.10e+03    
      31             44         7.8497e+03      1.93e+00       2.37e+01       9.41e+02    
      32             45         7.8476e+03      2.02e+00       7.66e+01       2.13e+03    
      33             46         7.8448e+03      2.86e+00       2.10e+02       1.38e+03    
      34             47         7.8401e+03      4.66e+00       5.13e+02       2.17e+03    
      35             48         7.8335e+03      6.64e+00       1.21e+03       9.41e+03    
      36             51         7.8323e+03      1.13e+00       1.50e+02       4.64e+03    
      37             52         7.8299e+03      2.45e+00       1.74e+02       7.10e+03    
      38             53         7.8271e+03      2.76e+00       4.67e+02       5.30e+03    
      39             56         7.8266e+03      4.86e-01       7.22e+01       9.83e+03    
`ftol` termination condition is satisfied.
Function evaluations 56, initial cost 1.8049e+05, final cost 7.8266e+03, first-order optimality 9.83e+03.
{(0, 1): (1000, array([ 5.66253828, 83.93624448]))}
error: 31.18, mu: 8.7, ratio: 0.206
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         2.2220e+05                                    3.59e+04    
       1              2         1.4813e+04      2.07e+05       5.78e+03       8.87e+03    
       2              3         6.2407e+03      8.57e+03       1.39e+03       1.34e+05    
       3              5         4.1824e+03      2.06e+03       6.05e+02       4.17e+04    
       4              6         3.7662e+03      4.16e+02       2.66e+02       8.66e+03    
       5              8         3.7075e+03      5.87e+01       1.76e+02       3.78e+03    
       6              9         3.6794e+03      2.81e+01       1.54e+02       9.38e+02    
       7             11         3.6629e+03      1.65e+01       8.76e+01       2.05e+03    
       8             12         3.6504e+03      1.25e+01       1.79e+02       2.07e+03    
       9             13         3.6315e+03      1.89e+01       4.75e+02       2.48e+03    
      10             15         3.6228e+03      8.69e+00       3.77e+02       1.85e+03    
      11             16         3.6079e+03      1.49e+01       4.88e+02       2.04e+03    
      12             18         3.6003e+03      7.54e+00       2.97e+02       2.02e+03    
      13             19         3.5875e+03      1.28e+01       3.89e+02       2.25e+03    
      14             20         3.5776e+03      9.92e+00       1.00e+03       3.62e+04    
      15             21         3.5512e+03      2.65e+01       5.55e+02       1.31e+04    
      16             23         3.5430e+03      8.15e+00       2.60e+02       1.54e+04    
      17             24         3.5339e+03      9.14e+00       5.39e+02       1.41e+04    
      18             26         3.5292e+03      4.72e+00       2.96e+02       1.63e+04    
      19             27         3.5238e+03      5.36e+00       6.48e+02       2.92e+04    
      20             29         3.5214e+03      2.39e+00       3.72e+02       3.09e+04    
      21             30         3.5187e+03      2.76e+00       8.75e+02       6.85e+04    
      22             31         3.5177e+03      9.87e-01       2.15e+02       7.15e+03    
      23             32         3.5174e+03      2.83e-01       1.05e+02       5.03e+03    
`ftol` termination condition is satisfied.
Function evaluations 32, initial cost 2.2220e+05, final cost 3.5174e+03, first-order optimality 5.03e+03.
{(0, 1): (1000, array([ 5.18745672, 86.31974844]))}
error: 29.56, mu: 5.2, ratio: 0.150
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         7.4716e+04                                    1.80e+04    
       1              2         7.1910e+03      6.75e+04       3.70e+03       1.87e+04    
       2              3         2.6736e+03      4.52e+03       1.02e+03       5.56e+05    
       3              4         1.8425e+03      8.31e+02       1.15e+03       4.58e+05    
       4              6         1.3178e+03      5.25e+02       2.43e+02       2.88e+05    
       5              7         1.2127e+03      1.05e+02       1.48e+02       2.29e+04    
       6              8         1.2081e+03      4.62e+00       3.74e+02       3.78e+04    
       7              9         1.2052e+03      2.92e+00       7.63e+02       1.34e+05    
       8             10         1.2013e+03      3.89e+00       1.36e+03       5.18e+04    
       9             11         1.1973e+03      4.00e+00       1.82e+03       3.65e+04    
      10             12         1.1880e+03      9.30e+00       1.06e+03       6.16e+04    
      11             13         1.1826e+03      5.41e+00       8.67e+02       5.40e+04    
      12             14         1.1754e+03      7.18e+00       1.51e+03       4.62e+04    
      13             15         1.1713e+03      4.04e+00       2.57e+03       2.51e+05    
      14             16         1.1645e+03      6.82e+00       1.29e+03       2.64e+05    
      15             17         1.1461e+03      1.84e+01       6.88e+02       2.23e+04    
      16             19         1.1332e+03      1.30e+01       4.03e+02       5.34e+04    
      17             20         1.1249e+03      8.24e+00       7.61e+02       6.82e+04    
      18             21         1.1138e+03      1.11e+01       1.50e+03       1.67e+05    
      19             22         1.0966e+03      1.72e+01       3.00e+03       2.18e+05    
      20             23         1.0882e+03      8.48e+00       3.13e+02       1.01e+05    
      21             24         1.0826e+03      5.56e+00       7.24e+02       2.01e+05    
      22             25         1.0811e+03      1.53e+00       1.88e+02       5.54e+04    
      23             26         1.0810e+03      3.66e-02       2.49e+02       4.88e+05    
      24             27         1.0794e+03      1.62e+00       5.48e+01       1.41e+05    
      25             28         1.0787e+03      6.91e-01       5.27e+01       4.30e+03    
      26             29         1.0781e+03      6.18e-01       1.10e+02       6.50e+03    
      27             30         1.0770e+03      1.10e+00       2.35e+02       6.23e+04    
      28             32         1.0766e+03      4.16e-01       6.65e+01       4.85e+03    
      29             33         1.0759e+03      7.05e-01       1.45e+02       7.28e+03    
      30             34         1.0747e+03      1.15e+00       2.76e+02       2.52e+04    
      31             36         1.0743e+03      4.04e-01       5.22e+01       2.97e+04    
      32             37         1.0739e+03      4.61e-01       9.79e+01       9.82e+03    
      33             38         1.0730e+03      8.96e-01       1.93e+02       4.99e+03    
      34             39         1.0712e+03      1.75e+00       3.87e+02       5.03e+03    
      35             40         1.0684e+03      2.86e+00       7.93e+02       4.94e+03    
      36             41         1.0684e+03      8.79e-03       1.19e+02       5.58e+04    
      37             42         1.0676e+03      7.53e-01       3.39e+01       1.89e+04    
      38             43         1.0673e+03      3.11e-01       3.85e+01       1.43e+04    
      39             44         1.0670e+03      3.17e-01       8.02e+01       3.90e+03    
      40             45         1.0664e+03      5.69e-01       1.53e+02       9.64e+03    
      41             47         1.0662e+03      1.76e-01       3.82e+01       4.24e+02    
      42             48         1.0660e+03      2.89e-01       7.36e+01       7.29e+02    
      43             49         1.0654e+03      5.71e-01       1.49e+02       3.50e+02    
      44             50         1.0643e+03      1.04e+00       2.95e+02       2.60e+03    
      45             52         1.0642e+03      1.56e-01       2.57e+01       6.59e+03    
      46             53         1.0640e+03      1.82e-01       4.97e+01       4.08e+03    
      47             54         1.0637e+03      3.56e-01       9.71e+01       1.76e+03    
      48             55         1.0629e+03      7.03e-01       1.95e+02       1.33e+02    
      49             56         1.0616e+03      1.34e+00       3.89e+02       5.09e+03    
      50             57         1.0616e+03      7.27e-03       1.50e+02       1.11e+05    
      51             58         1.0611e+03      4.85e-01       3.12e+01       2.08e+04    
      52             59         1.0610e+03      1.49e-01       3.46e+01       2.55e+03    
      53             61         1.0609e+03      7.08e-02       1.86e+01       3.17e+03    
`ftol` termination condition is satisfied.
Function evaluations 61, initial cost 7.4716e+04, final cost 1.0609e+03, first-order optimality 3.17e+03.
{(0, 1): (1000, array([ 4.27818178, 84.48084784]))}
error: 27.24, mu: 4.3, ratio: 0.150
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         1.5596e+05                                    3.00e+04    
       1              2         2.1541e+04      1.34e+05       5.40e+03       1.86e+05    
       2              4         5.1724e+03      1.64e+04       4.97e+02       1.58e+05    
       3              5         1.9094e+03      3.26e+03       3.58e+02       2.51e+04    
       4              6         1.4257e+03      4.84e+02       2.93e+02       1.01e+04    
       5              8         1.1964e+03      2.29e+02       1.05e+02       1.12e+04    
       6              9         1.1215e+03      7.49e+01       2.43e+02       1.92e+03    
       7             11         1.0571e+03      6.44e+01       7.53e+01       1.32e+03    
       8             12         1.0282e+03      2.89e+01       1.69e+02       4.09e+02    
       9             14         1.0230e+03      5.20e+00       2.08e+02       7.58e+02    
      10             15         1.0144e+03      8.63e+00       3.68e+02       2.17e+03    
      11             16         1.0017e+03      1.27e+01       9.55e+02       4.29e+02    
      12             17         9.7100e+02      3.07e+01       1.61e+03       9.12e+03    
      13             19         9.4560e+02      2.54e+01       3.09e+02       2.81e+03    
      14             20         9.4188e+02      3.72e+00       4.16e+02       1.87e+04    
      15             21         9.0079e+02      4.11e+01       3.59e+01       2.50e+03    
      16             22         8.9481e+02      5.98e+00       3.69e+01       2.08e+03    
      17             23         8.8999e+02      4.83e+00       5.42e+01       5.24e+03    
      18             24         8.8330e+02      6.68e+00       5.93e+01       2.49e+03    
      19             26         8.8123e+02      2.07e+00       1.70e+01       5.46e+02    
      20             27         8.7922e+02      2.01e+00       3.34e+01       3.40e+03    
      21             29         8.7780e+02      1.42e+00       1.69e+01       1.54e+03    
      22             30         8.7775e+02      5.26e-02       3.31e+01       1.95e+04    
      23             31         8.7545e+02      2.30e+00       7.47e+00       3.12e+02    
      24             32         8.7515e+02      3.05e-01       6.97e+00       2.31e+03    
      25             33         8.7479e+02      3.52e-01       1.47e+01       1.35e+04    
      26             34         8.7424e+02      5.57e-01       9.82e+00       4.21e+04    
      27             35         8.7371e+02      5.22e-01       2.96e+01       2.79e+03    
      28             36         8.7329e+02      4.24e-01       3.09e+01       3.91e+03    
      29             37         8.7259e+02      6.97e-01       2.42e+01       2.99e+03    
      30             38         8.7181e+02      7.85e-01       1.37e+01       1.81e+03    
      31             40         8.7142e+02      3.88e-01       5.85e+00       1.42e+03    
      32             41         8.7130e+02      1.16e-01       1.14e+01       6.28e+03    
      33             42         8.7074e+02      5.69e-01       4.05e+00       1.69e+02    
      34             43         8.7064e+02      9.82e-02       3.01e+00       5.64e+02    
      35             44         8.7049e+02      1.46e-01       5.68e+00       2.30e+03    
      36             45         8.7047e+02      2.17e-02       5.28e+00       1.21e+02    
`ftol` termination condition is satisfied.
Function evaluations 45, initial cost 1.5596e+05, final cost 8.7047e+02, first-order optimality 1.21e+02.
{(0, 1): (1000, array([ 2.70283707, 42.42796182]))}
error: 15.19, mu: 2.7, ratio: 0.150
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         6.1437e+04                                    1.26e+04    
       1              2         1.8430e+04      4.30e+04       3.29e+03       1.96e+03    
       2              3         1.9672e+03      1.65e+04       1.91e+03       3.06e+04    
       3              6         1.2951e+03      6.72e+02       2.67e+02       1.86e+04    
       4              7         9.1192e+02      3.83e+02       2.46e+02       5.68e+03    
       5              8         7.0882e+02      2.03e+02       1.91e+02       5.75e+03    
       6             10         7.0102e+02      7.80e+00       4.75e+01       3.78e+03    
       7             11         6.9405e+02      6.96e+00       9.50e+01       2.43e+03    
       8             12         6.8308e+02      1.10e+01       1.15e+02       1.11e+03    
       9             13         6.7212e+02      1.10e+01       1.17e+02       1.51e+03    
      10             14         6.5745e+02      1.47e+01       2.35e+02       3.81e+03    
      11             15         6.4223e+02      1.52e+01       3.97e+02       1.17e+04    
      12             16         6.0707e+02      3.52e+01       3.15e+02       9.29e+03    
      13             18         5.8921e+02      1.79e+01       9.22e+01       2.04e+03    
      14             19         5.8373e+02      5.49e+00       1.61e+02       1.90e+04    
      15             20         5.5812e+02      2.56e+01       1.56e+02       1.79e+04    
      16             21         5.5353e+02      4.59e+00       9.23e+01       2.49e+04    
      17             22         5.3649e+02      1.70e+01       1.66e+01       2.17e+02    
      18             23         5.3534e+02      1.15e+00       1.15e+01       1.70e+02    
      19             24         5.3367e+02      1.67e+00       2.76e+01       1.15e+03    
      20             25         5.3147e+02      2.20e+00       5.66e+01       3.71e+03    
      21             27         5.2991e+02      1.56e+00       2.69e+01       1.41e+03    
      22             28         5.2795e+02      1.96e+00       5.05e+01       3.05e+03    
      23             30         5.2682e+02      1.13e+00       2.77e+01       1.43e+03    
      24             31         5.2555e+02      1.27e+00       5.26e+01       3.90e+03    
      25             32         5.2412e+02      1.43e+00       5.03e+01       2.77e+03    
      26             33         5.2289e+02      1.23e+00       5.28e+01       2.58e+03    
      27             35         5.2226e+02      6.26e-01       2.88e+01       9.97e+02    
      28             36         5.2159e+02      6.72e-01       5.58e+01       2.98e+03    
      29             37         5.2104e+02      5.52e-01       5.95e+01       3.51e+03    
      30             38         5.2040e+02      6.43e-01       6.18e+01       3.89e+03    
      31             39         5.1975e+02      6.48e-01       5.80e+01       3.21e+03    
      32             41         5.1930e+02      4.44e-01       3.01e+01       1.28e+03    
      33             42         5.1929e+02      1.60e-02       5.70e+01       3.43e+03    
      34             43         5.1890e+02      3.92e-01       4.78e+00       2.53e+01    
      35             44         5.1882e+02      7.50e-02       1.36e+01       2.25e+02    
      36             45         5.1871e+02      1.07e-01       2.82e+01       7.99e+02    
      37             46         5.1860e+02      1.16e-01       1.67e+01       3.31e+02    
      38             47         5.1855e+02      4.32e-02       5.11e+00       2.54e+01    
`ftol` termination condition is satisfied.
Function evaluations 47, initial cost 6.1437e+04, final cost 5.1855e+02, first-order optimality 2.54e+01.
{(0, 1): (1000, array([ 1.77068505, 26.61602651]))}
error: 10.18, mu: 1.8, ratio: 0.150
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         5.9865e+04                                    1.06e+04    
       1              2         6.2718e+03      5.36e+04       3.43e+03       2.27e+03    
       2              3         4.3888e+02      5.83e+03       9.63e+02       1.67e+04    
       3              5         2.0774e+02      2.31e+02       4.44e+02       2.70e+04    
       4              6         1.3757e+02      7.02e+01       1.84e+02       8.80e+02    
       5              8         1.3334e+02      4.23e+00       2.01e+02       2.99e+03    
       6              9         1.3217e+02      1.18e+00       3.43e+02       3.50e+03    
       7             10         1.2901e+02      3.16e+00       1.51e+02       1.68e+03    
       8             11         1.2883e+02      1.81e-01       1.76e+02       4.72e+03    
       9             12         1.2740e+02      1.42e+00       1.73e+01       4.04e+01    
      10             13         1.2730e+02      1.08e-01       2.55e+01       3.20e+02    
      11             14         1.2720e+02      9.54e-02       5.07e+01       1.46e+03    
      12             15         1.2706e+02      1.39e-01       4.49e+01       1.50e+03    
      13             16         1.2696e+02      1.00e-01       4.03e+01       3.96e+02    
      14             17         1.2695e+02      1.46e-02       1.90e+01       6.14e+01    
      15             18         1.2694e+02      7.38e-03       2.43e+01       5.52e+01    
`ftol` termination condition is satisfied.
Function evaluations 18, initial cost 5.9865e+04, final cost 1.2694e+02, first-order optimality 5.52e+01.
{(0, 1): (1000, array([ 1.70109626, 22.84273331]))}
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         8.9060e+08                                    3.51e+05    
       1              2         1.0372e+08      7.87e+08       2.54e+05       1.51e+05    
       2              3         6.7810e+05      1.03e+08       1.21e+05       2.94e+04    
       3              5         3.1674e+05      3.61e+05       6.48e+03       2.37e+06    
       4              6         9.1654e+04      2.25e+05       5.12e+03       6.38e+04    
       5              8         5.8256e+04      3.34e+04       1.17e+03       2.45e+04    
       6              9         5.1142e+04      7.11e+03       1.03e+03       1.51e+04    
       7             11         4.3991e+04      7.15e+03       6.87e+02       3.58e+04    
       8             14         4.3628e+04      3.63e+02       5.71e+01       2.43e+04    
       9             15         4.3388e+04      2.40e+02       7.42e+01       2.21e+04    
      10             16         4.3174e+04      2.14e+02       9.55e+01       2.12e+04    
      11             17         4.3007e+04      1.67e+02       1.26e+02       2.13e+04    
      12             19         4.2815e+04      1.92e+02       2.09e+01       2.18e+04    
      13             20         4.2692e+04      1.22e+02       4.35e+01       2.18e+04    
      14             21         4.2548e+04      1.44e+02       7.97e+01       2.17e+04    
      15             22         4.2463e+04      8.53e+01       7.67e+01       2.17e+04    
      16             23         4.2459e+04      3.33e+00       9.54e+01       2.18e+04    
      17             24         4.2294e+04      1.65e+02       1.43e+01       2.18e+04    
      18             25         4.2203e+04      9.10e+01       4.38e+01       2.18e+04    
      19             26         4.2119e+04      8.44e+01       2.55e+01       2.18e+04    
      20             27         4.2039e+04      7.96e+01       2.85e+01       2.18e+04    
      21             28         4.1893e+04      1.46e+02       4.87e+01       2.18e+04    
      22             29         4.1740e+04      1.53e+02       1.24e+02       2.19e+04    
      23             31         4.1577e+04      1.63e+02       2.01e+01       2.26e+04    
      24             32         4.1469e+04      1.08e+02       3.47e+01       2.25e+04    
      25             33         4.1345e+04      1.24e+02       7.58e+01       2.24e+04    
      26             34         4.1287e+04      5.80e+01       8.12e+01       2.25e+04    
      27             35         4.1180e+04      1.07e+02       1.14e+01       2.26e+04    
      28             36         4.1094e+04      8.59e+01       4.30e+01       2.26e+04    
      29             37         4.0935e+04      1.60e+02       2.89e+01       2.26e+04    
      30             38         4.0654e+04      2.80e+02       8.51e+01       2.25e+04    
      31             39         4.0557e+04      9.73e+01       2.12e+02       2.15e+04    
      32             40         4.0328e+04      2.29e+02       3.78e+01       2.34e+04    
      33             41         4.0281e+04      4.63e+01       3.69e+01       2.32e+04    
      34             42         4.0031e+04      2.50e+02       1.87e+01       2.36e+04    
      35             43         3.9891e+04      1.40e+02       5.91e+01       2.40e+04    
      36             44         3.9778e+04      1.13e+02       4.06e+01       2.39e+04    
      37             45         3.9661e+04      1.17e+02       7.93e+01       2.38e+04    
      38             46         3.9608e+04      5.26e+01       8.13e+01       2.38e+04    
      39             47         3.9501e+04      1.07e+02       1.17e+01       2.40e+04    
      40             48         3.9417e+04      8.37e+01       4.36e+01       2.40e+04    
      41             49         3.9261e+04      1.56e+02       2.95e+01       2.39e+04    
      42             50         3.8985e+04      2.75e+02       8.64e+01       2.40e+04    
      43             51         3.8878e+04      1.08e+02       2.13e+02       2.29e+04    
      44             52         3.8649e+04      2.28e+02       3.80e+01       2.52e+04    
      45             53         3.8592e+04      5.74e+01       3.86e+01       2.48e+04    
      46             54         3.8353e+04      2.39e+02       1.88e+01       2.53e+04    
      47             55         3.8218e+04      1.35e+02       5.93e+01       2.57e+04    
      48             56         3.8107e+04      1.11e+02       3.89e+01       2.55e+04    
      49             57         3.7984e+04      1.23e+02       7.59e+01       2.53e+04    
      50             58         3.7914e+04      6.97e+01       7.76e+01       2.54e+04    
      51             59         3.7912e+04      2.32e+00       9.22e+01       2.55e+04    
      52             60         3.7754e+04      1.58e+02       1.55e+01       2.56e+04    
      53             61         3.7667e+04      8.69e+01       4.44e+01       2.55e+04    
      54             62         3.7586e+04      8.02e+01       2.40e+01       2.54e+04    
      55             63         3.7450e+04      1.37e+02       5.21e+01       2.55e+04    
      56             64         3.7357e+04      9.29e+01       7.59e+01       2.57e+04    
      57             65         3.7330e+04      2.70e+01       8.92e+01       2.58e+04    
      58             66         3.7196e+04      1.34e+02       1.38e+01       2.60e+04    
      59             67         3.7111e+04      8.55e+01       4.35e+01       2.60e+04    
      60             68         3.6959e+04      1.52e+02       3.61e+01       2.60e+04    
      61             69         3.6731e+04      2.28e+02       1.08e+02       2.62e+04    
      62             71         3.6616e+04      1.14e+02       2.48e+01       2.78e+04    
      63             72         3.6523e+04      9.34e+01       2.28e+01       2.75e+04    
      64             73         3.6387e+04      1.36e+02       6.22e+01       2.74e+04    
      65             74         3.6288e+04      9.86e+01       7.31e+01       2.76e+04    
      66             75         3.6248e+04      4.06e+01       8.77e+01       2.76e+04    
      67             76         3.6119e+04      1.29e+02       1.40e+01       2.79e+04    
      68             77         3.6030e+04      8.88e+01       4.17e+01       2.79e+04    
      69             78         3.5870e+04      1.60e+02       3.47e+01       2.79e+04    
      70             79         3.5595e+04      2.74e+02       9.72e+01       2.86e+04    
      71             81         3.5493e+04      1.02e+02       2.49e+01       3.06e+04    
      72             82         3.5402e+04      9.20e+01       1.87e+01       3.03e+04    
      73             83         3.5248e+04      1.54e+02       5.33e+01       3.02e+04    
      74             84         3.4984e+04      2.63e+02       1.03e+02       3.10e+04    
      75             86         3.4869e+04      1.16e+02       2.47e+01       3.33e+04    
      76             87         3.4768e+04      1.00e+02       2.17e+01       3.30e+04    
      77             88         3.4606e+04      1.62e+02       5.46e+01       3.28e+04    
      78             89         3.4316e+04      2.90e+02       9.90e+01       3.42e+04    
      79             91         3.4199e+04      1.17e+02       2.39e+01       3.71e+04    
      80             92         3.4093e+04      1.06e+02       2.05e+01       3.69e+04    
      81             93         3.3913e+04      1.80e+02       4.98e+01       3.67e+04    
      82             94         3.3569e+04      3.44e+02       9.01e+01       3.94e+04    
      83             95         3.3114e+04      4.55e+02       2.05e+02       4.43e+04    
      84             97         3.2851e+04      2.63e+02       3.91e+01       5.24e+04    
      85             98         3.2584e+04      2.67e+02       3.89e+01       5.29e+04    
      86             99         3.2372e+04      2.12e+02       7.67e+01       5.32e+04    
      87             100        3.2111e+04      2.61e+02       4.26e+01       5.23e+04    
      88             101        3.1857e+04      2.54e+02       6.55e+01       5.13e+04    
      89             102        3.1565e+04      2.92e+02       4.44e+01       4.86e+04    
      90             103        3.1267e+04      2.98e+02       5.51e+01       4.39e+04    
      91             104        3.0953e+04      3.13e+02       4.14e+01       3.64e+04    
      92             105        3.0431e+04      5.23e+02       9.91e+01       2.58e+04    
      93             106        3.0034e+04      3.96e+02       2.30e+02       2.06e+04    
      94             108        2.9471e+04      5.64e+02       2.07e+01       1.59e+04    
      95             109        2.9112e+04      3.59e+02       7.01e+01       1.02e+04    
      96             110        2.8913e+04      1.99e+02       8.32e+01       7.51e+03    
      97             111        2.8693e+04      2.20e+02       3.86e+01       6.26e+03    
      98             112        2.8512e+04      1.81e+02       6.83e+01       5.76e+03    
      99             113        2.8329e+04      1.83e+02       5.11e+01       5.24e+03    
      100            114        2.8163e+04      1.66e+02       6.47e+01       4.78e+03    
      101            115        2.8004e+04      1.59e+02       5.90e+01       4.34e+03    
      102            116        2.7858e+04      1.46e+02       6.71e+01       5.27e+03    
      103            117        2.7722e+04      1.36e+02       6.55e+01       6.57e+03    
      104            118        2.7599e+04      1.23e+02       7.17e+01       7.71e+03    
      105            119        2.7488e+04      1.12e+02       7.09e+01       8.75e+03    
      106            120        2.7387e+04      1.00e+02       7.52e+01       9.62e+03    
      107            121        2.7295e+04      9.21e+01       7.36e+01       1.04e+04    
      108            122        2.7210e+04      8.50e+01       7.61e+01       1.11e+04    
      109            123        2.7083e+04      1.27e+02       2.09e+01       1.12e+04    
      110            124        2.6992e+04      9.12e+01       4.57e+01       1.16e+04    
      111            125        2.6843e+04      1.48e+02       4.60e+01       1.21e+04    
      112            126        2.6576e+04      2.67e+02       1.04e+02       1.24e+04    
      113            127        2.6116e+04      4.60e+02       2.38e+02       1.27e+04    
      114            128        2.5393e+04      7.24e+02       5.16e+02       1.38e+04    
      115            130        2.5084e+04      3.09e+02       2.79e+02       9.28e+03    
      116            132        2.4981e+04      1.02e+02       4.89e+01       1.19e+04    
      117            133        2.4961e+04      2.08e+01       8.45e+01       1.17e+04    
      118            134        2.4807e+04      1.54e+02       2.22e+01       1.18e+04    
      119            135        2.4726e+04      8.06e+01       5.65e+01       1.21e+04    
      120            136        2.4665e+04      6.17e+01       3.20e+01       1.23e+04    
      121            137        2.4561e+04      1.04e+02       6.33e+01       1.24e+04    
      122            138        2.4367e+04      1.94e+02       1.30e+02       1.24e+04    
      123            139        2.4049e+04      3.18e+02       2.74e+02       1.23e+04    
      124            141        2.3956e+04      9.28e+01       6.91e+01       1.12e+04    
      125            142        2.3935e+04      2.13e+01       8.01e+01       1.22e+04    
      126            143        2.3837e+04      9.77e+01       1.90e+01       1.22e+04    
      127            144        2.3781e+04      5.63e+01       6.10e+01       1.22e+04    
      128            145        2.3729e+04      5.16e+01       2.99e+01       1.23e+04    
      129            146        2.3640e+04      8.97e+01       6.90e+01       1.23e+04    
      130            147        2.3466e+04      1.73e+02       1.36e+02       1.21e+04    
      131            148        2.3152e+04      3.14e+02       2.86e+02       1.19e+04    
      132            149        2.2622e+04      5.31e+02       5.92e+02       1.19e+04    
      133            151        2.2363e+04      2.58e+02       3.12e+02       8.86e+03    
      134            153        2.2299e+04      6.42e+01       6.38e+01       1.06e+04    
      135            155        2.2210e+04      8.87e+01       1.61e+01       1.00e+04    
      136            156        2.2160e+04      5.00e+01       5.88e+01       1.03e+04    
      137            157        2.2083e+04      7.70e+01       7.04e+01       1.03e+04    
      138            158        2.1939e+04      1.44e+02       1.48e+02       1.02e+04    
      139            159        2.1678e+04      2.61e+02       3.01e+02       9.94e+03    
      140            160        2.1377e+04      3.02e+02       6.24e+02       9.91e+03    
      141            163        2.1152e+04      2.25e+02       2.88e+01       8.12e+03    
      142            164        2.1083e+04      6.83e+01       7.73e+01       9.03e+03    
      143            165        2.1010e+04      7.34e+01       7.46e+01       8.85e+03    
      144            166        2.0875e+04      1.35e+02       1.50e+02       8.98e+03    
      145            167        2.0639e+04      2.37e+02       3.13e+02       8.01e+03    
      146            168        2.0247e+04      3.92e+02       6.60e+02       7.29e+03    
      147            169        1.9620e+04      6.26e+02       1.44e+03       8.80e+03    
      148            170        1.8532e+04      1.09e+03       2.82e+03       1.90e+04    
      149            171        1.6820e+04      1.71e+03       4.89e+03       9.20e+04    
      150            173        1.6399e+04      4.21e+02       1.76e+03       2.51e+04    
      151            174        1.4653e+04      1.75e+03       2.44e+03       5.56e+04    
      152            176        1.4203e+04      4.50e+02       5.05e+02       1.96e+04    
      153            179        1.4066e+04      1.37e+02       7.61e+01       1.69e+04    
      154            181        1.3996e+04      6.99e+01       2.57e+01       1.42e+04    
      155            182        1.3962e+04      3.36e+01       5.69e+01       8.68e+03    
      156            183        1.3928e+04      3.41e+01       5.62e+01       8.75e+03    
      157            184        1.3869e+04      5.85e+01       8.89e+01       6.59e+03    
      158            185        1.3784e+04      8.57e+01       1.11e+02       7.84e+03    
      159            186        1.3656e+04      1.28e+02       2.25e+02       9.22e+03    
      160            187        1.3453e+04      2.03e+02       7.14e+02       4.59e+03    
      161            188        1.3302e+04      1.51e+02       2.27e+03       6.32e+03    
      162            190        1.3083e+04      2.19e+02       4.18e+02       1.39e+04    
      163            191        1.2697e+04      3.87e+02       7.53e+02       5.32e+04    
      164            192        1.2605e+04      9.18e+01       1.80e+03       2.01e+05    
      165            193        1.2090e+04      5.15e+02       3.38e+02       2.49e+04    
      166            195        1.2005e+04      8.43e+01       1.70e+02       9.20e+03    
      167            198        1.1986e+04      1.93e+01       1.58e+01       1.29e+04    
      168            199        1.1979e+04      6.55e+00       4.31e+01       8.34e+02    
      169            200        1.1979e+04      0.00e+00       0.00e+00       8.34e+02    
The maximum number of function evaluations is exceeded.
Function evaluations 200, initial cost 8.9060e+08, final cost 1.1979e+04, first-order optimality 8.34e+02.
{(0, 1): (1000, array([0.83903596, 5.43167498]))}
error:  3.100034746360199
/usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)

[ericleonardis]

Here is an update about progress on debugging this issue.

I have been able to replicate the error on my side using different datasets, as well. It is unclear whether this bug is an indication of bad detections or if it is specifically a plotting error. The error seems quite low on the calibration, which indicates that the calibration file I am currently going through the make_reproj_imgs() function to make sure that the frames are aligned with the detection and reprojection labels. It is possible that it is plotting the detections and reprojections from different images.

First issue is that the calibration images are never being generated. I had to generate them separately in ffmpeg before the reprojection images could be generated. I will make a pull request that uses an ffmpeg command line interface to render the images. It would be more time efficient to sample the images randomly then only render the images which are being sampled, rather than the whole video itself.

So far the list that reads in the images list(cam.glob(f"/{frame}.jpg"))[0] correctly references the calibration images like "frame_0001.jpg" etc. This suggests to me that the detections and reprojections might be misaligned. Next I will be investigating whether reprojections[i, frames.index(frame), :, 0] and detections[i, frames.index(frame), :, 0] is properly aligned to those images.

I will continue tracking down the bug and report back on how the debugging process is going.

[delaroob]

Hi! Any updates on this?

[delaroob]

Hi @ericleonardis, any updates on this? :)

[ericleonardis]

@delaroob Sorry for the delay, but problem solved!

The problem in Issue #41 was caused by a mismatch between the number of frames in the video and the length of the frames list input into the make_reproj_imgs() function. In my tests, I noticed that earlier sample frames seemed to match, while later sampled frames did not match. This is because each of the frames that lead to no detections would be subtracted from the list of total frames. So they become progressively more mismatched. The good news however is that the calibration itself was not compromised by this bug just the plotting. So calibration files should be good to go, and histograms remain an accurate representation of the error.

I have written a new version of the make_reproj_imgs() function which grabs the frame from the video using imageio and ffmpeg rather than from a list of images. The eric/calibration-imgs branch eliminates the problem of misaligned detections, reprojections and images and has is the new PR #42 .

The quickest way to fix it for yourself right now is to check out my branch eric/calibration-imgs at https://github.com/talmolab/sleap-anipose/blob/9c7f7a18561035b84105306e0e202ef468e8263c/sleap_anipose/calibration.py#L66 or swap out the make_reproj_imgs() function above. Once my PR #42 is merged, then it should also work from the regular pip install.

Thank you for your patience.