Mask and indexed tensor shapes do not match.

[marybethcassity]

Hi! When trying to train a model, I get this error right after I see the Sanity Checking DataLoader: IndexError: The shape of the mask [32, 40] at index 1 does not match the shape of the indexed tensor [32, 20, 64, 64] at index 1. As far as I can tell this is a mismatch with the amount of keypoints. I changed the config to reflect this (num_keypoints: 40) but am a bit confused. I only tracked 20 bodyparts with my DLC model. What am I missing? Is this bodyparts*2 for the x and y coordinate? Please let me know if it would be helpful for me to post the entire error or what info would be helpful.

[themattinthehatt]

@marybethcassity the config should contain the actual number of bodyparts/keypoints (20) and yes, later the fact that each keypoint has an x and y coordinate will be taken into account automatically by the code. Would you mind sharing the data portion of your config file? Also, can you double check that your labeled data csv file does in fact have 20 keypoints?

[marybethcassity]

Here is what I have in my config file. I have num_keypoints: 40 because num_keypoints: 20 got the error.

data:
  # dimensions of training images
  image_orig_dims:
    height: 2160
    width: 3840
  # resize dimensions to streamline model creation
  image_resize_dims:
    height: 256
    width: 256
  # ABSOLUTE path to data directory
  data_dir: /mnt/lightning_pose/RatBody_Combined-MB-2024-04-29
  # ABSOLUTE path to unlabeled videos' directory
  video_dir: /mnt/lightning_pose/RatBody_Combined-MB-2024-04-29/videos
  # location of labels; this should be relative to `data_dir`
  csv_file: /mnt/lightning_pose/RatBody_Combined-MB-2024-04-29/CollectedData.csv
  # downsample heatmaps - 2 | 3
  downsample_factor: 2
  # total number of keypoints
  num_keypoints: 40
  # keypoint names
  keypoint_names: 
    - nose
    - left_eye
    - left_ear
    - right_eye
    - right_ear
    - camera_frontleft
    - camera_backleft
    - camera_frontright
    - camera_backright
    - left_elbow
    - left_paw
    - right_elbow
    - right_paw
    - middleback
    - left_knee
    - left_hindpaw
    - right_knee
    - right_hindpaw
    - tailbase
    - tailend
  # for mirrored setups with all keypoints defined in same csv file, define matching
  # columns for different keypoints (assumes x-y-x-y interleaving)
  # each list corresponds to a single view, so in the example below there are 2 views
  # keypoint 0 is from view 0 and matches up with keypoint 8 from view 2
  # columns that correspond to keypoints only labeled in a single view are omitted
  # this info is only used for the multiview pca loss
  mirrored_column_matches: null
  # list of indices of keypoints used for pca singleview loss (use order of labels file)
  columns_for_singleview_pca: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

My CollectedData csv does have 20 keypoints. Here is a snort snippet of it.

bodyparts,nose,nose,left_eye,left_eye,left_ear,left_ear,right_eye,right_eye,right_ear,right_ear,camera_frontleft,camera_frontleft,camera_backleft,camera_backleft,camera_frontright,camera_frontright,camera_backright,camera_backright,left_elbow,left_elbow,left_paw,left_paw,right_elbow,right_elbow,right_paw,right_paw,middleback,middleback,left_knee,left_knee,left_hindpaw,left_hindpaw,right_knee,right_knee,right_hindpaw,right_hindpaw,tailbase,tailbase,tailend,tailend,nose,nose,left_eye,left_eye,left_ear,left_ear,right_eye,right_eye,right_ear,right_ear,camera_frontleft,camera_frontleft,camera_backleft,camera_backleft,camera_frontright,camera_frontright,camera_backright,camera_backright,left_elbow,left_elbow,left_paw,left_paw,right_elbow,right_elbow,right_paw,right_paw,middleback,middleback,left_knee,left_knee,left_hindpaw,left_hindpaw,right_knee,right_knee,right_hindpaw,right_hindpaw,tailbase,tailbase,tailend,tailend
coords,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y,x,y
labeled-data/20231127_Front_Short/img008.png,1768.7853274864842,2014.8738009101403,1799.336669921875,1978.594482421875,1835.271728515625,1945.193603515625,,,,,1787.7421875,1908.576171875,1821.205078125,1879.5877685546875,1772.1304931640625,1887.6434326171875,1798.22607421875,1863.3265380859373,1929.4333713526585,1987.3012559173192,1891.10205078125,2010.028564453125,,,1822.6187744140625,2019.320068359375,1937.4219970703125,1850.09619140625,2016.3231201171875,1947.722412109375,2003.85595703125,1997.0579833984373,,,1993.757080078125,1992.89111328125,2094.67333984375,1918.7501220703125,2350.3037109375,1833.8302001953125,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
labeled-data/20231127_Front_Short/img009.png,1761.3059391228269,2015.382352904305,1800.539701755988,1980.1933080683768,1838.56004904998,1948.2400374702347,,,,,1792.164031400894,1911.5363067932865,1826.337975371192,1882.2622447873457,1776.0352171304055,1890.8979114381243,1803.0216754140893,1865.206803152057,1952.1994393162904,1962.56705965216,1892.4855347576809,2014.4066031700736,,,1834.2254354624,2016.1092996090888,1943.063374105281,1848.3695372782197,2012.3727115707093,1931.42009431012,2006.30563487486,2000.5847686428065,,,,,2097.2104391754133,1919.2142961191007,2353.249094691285,1835.7687350971617,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

I did, however use python scripts/converters/dlc2lp.py --dlc_dir=/path/to/dlc_dir --lp_dir=/path/to/lp_dir to create the project before I realized that this only works for installs through conda (I used Docker). Let me copy CollectedData.csv again and see if that fixes it.

EDIT: The csv does not have 20 keypoints!-- I don't know how I missed that! I'll fix that and try again!

[marybethcassity]

@themattinthehatt Here is what I think happened. We had two labelers for our original DLC project. (Instead of labeling in the same project, I made two different DLC projects, then combined the datasets later.) When I ran python scripts/converters/dlc2lp.py --dlc_dir=/path/to/dlc_dir --lp_dir=/path/to/lp_dir the output looked like this (scroll to the right and down to see what I'm talking about). I've had other issues in the past with having two different labelers. I manually fixed it for now, but will write a script to automate it and can share if that would be helpful for others!

[themattinthehatt]

Yes that makes total sense - glad you found the fix! Please feel free to share the script once you're done :)