search

mprib / caliscope

Multicamera Calibration + Pose Estimation --> Open Source Motion Capture
https://mprib.github.io/caliscope/
BSD 2-Clause "Simplified" License
190 stars 10 forks source link

Support output to OpenPose JSON format #650

Open mprib opened 1 day ago

mprib commented 1 day ago

OpenPose provides a standard JSON format for its output:

https://cmu-perceptual-computing-lab.github.io/openpose/web/html/doc/md_doc_02_output.html

Rather than attempting to create some new style of representation, it may be best to default to a previously existing format that has survived some tire-kicking.

Note that OpenPose links to a format for representing the camera matrix, but the link is currently 404'd: https://cmu-perceptual-computing-lab.github.io/openpose/web/html/doc/advanced/calibration_module.md

davidpagnon commented 1 day ago

Two other standards are:

I can send you samples of OpenPose, deeplabcut, and Coco files if you are interested.

mprib commented 1 day ago

@davidpagnon,

Thank you. I am interested so if those are easily accessible it would be awesome to have some references.

Mac

davidpagnon commented 1 day ago

Here are some sample data (not from the same trials):

OpenPose format (one file per frame, keypoint order depends on model, coordinates x,y,conf) ```json { "people": [ { "face_keypoints_2d": [ ], "face_keypoints_3d": [ ], "hand_left_keypoints_2d": [ ], "hand_left_keypoints_3d": [ ], "hand_right_keypoints_2d": [ ], "hand_right_keypoints_3d": [ ], "person_id": [ -1 ], "pose_keypoints_2d": [ 785.032272338867, 874.429085731506, 0.930248200893402, 796.808270772298, 862.653086662292, 0.927412450313568, 799.752270380656, 862.653086662292, 0.861262261867523, 814.472268422445, 895.037084102631, 0.902777314186096, 902.792256673177, 903.869083404541, 0.949171543121338, 743.816277821859, 1051.06907176971, 0.765520691871643, 935.176252365112, 1036.3490729332, 0.809015989303589, 702.60028330485, 1227.70905780792, 0.772481501102448, 905.736256281535, 1207.1010594368, 0.780025482177734, 605.448296229044, 1333.69304943085, 0.793694794178009, 835.080265680949, 1260.09305524826, 0.558070302009583, 699.656283696493, 1339.58104896545, 0.680959105491638, 808.584269205729, 1342.52504873276, 0.684390842914581, 640.776291529337, 1525.05303430557, 0.8237544298172, 746.760277430217, 1486.78103733063, 0.782606363296509, 581.896299362183, 1692.86102104187, 0.868499636650085, 705.544282913208, 1648.70102453232, 0.813148558139801, 855.688262939453, 794.941092014313, 0.893165171146393, 849.800263722738, 962.74907875061, 0.913672804832458, 761.480275472005, 1321.91705036163, 1.05537974834442, 508.296309153239, 1636.9250254631, 0.869200706481934, 652.552289962769, 1616.31702709198, 0.742750108242035, 502.408309936523, 1666.36502313614, 0.921308159828186, 687.880285263062, 1628.09302616119, 0.713791012763977, 584.840298970541, 1728.18901824951, 0.935503244400024, 705.544282913208, 1675.19702243805, 0.864744603633881 ], "pose_keypoints_3d": [ ] }, { "face_keypoints_2d": [ ], "face_keypoints_3d": [ ], "hand_left_keypoints_2d": [ ], "hand_left_keypoints_3d": [ ], "hand_right_keypoints_2d": [ ], "hand_right_keypoints_3d": [ ], "person_id": [ -1 ], "pose_keypoints_2d": [ 493.176073074341, 434.241302490234, 0.719350218772888, 503.458368301392, 421.388433456421, 0.783019661903381, 480.323204040527, 421.388433456421, 0.755906939506531, 511.17008972168, 423.959007263184, 0.819137573242188, 444.33517074585, 413.676712036133, 0.867408633232117, 521.45238494873, 501.076221466064, 0.770452260971069, 369.788530349731, 485.652778625488, 0.772427380084991, 539.446401596069, 603.899173736572, 0.775422871112823, 287.530168533325, 596.187452316284, 0.830751299858093, 549.72869682312, 709.292699813843, 0.765416979789734, 261.824430465698, 719.574995040894, 0.820626974105835, 470.040908813477, 732.427864074707, 0.74791944026947, 377.50025177002, 727.286716461182, 0.713379383087158, 457.188039779663, 912.368030548096, 0.840719163417816, 364.647382736206, 902.085735321045, 0.842771649360657, 421.200006484985, 1087.16704940796, 0.855790734291077, 333.800497055054, 1007.47926139832, 0.843364059925079, 500.887794494629, 339.130071640015, 0.81070351600647, 457.188039779663, 462.517614364624, 0.845634043216705, 423.770580291748, 724.716142654419, 1.06806135177612, 436.623449325562, 1151.43139457703, 0.824704349040985, 341.512218475342, 1069.17303276062, 0.773277282714844, 452.046892166138, 1143.71967315674, 0.841744661331177, 318.377054214478, 1058.89073753357, 0.780112028121948, 403.205989837646, 1105.1610660553, 0.787861347198486, 328.659349441528, 1010.04983520508, 0.746102213859558 ], "pose_keypoints_3d": [ ] }, { "face_keypoints_2d": [ ], "face_keypoints_3d": [ ], "hand_left_keypoints_2d": [ ], "hand_left_keypoints_3d": [ ], "hand_right_keypoints_2d": [ ], "hand_right_keypoints_3d": [ ], "person_id": [ -1 ], "pose_keypoints_2d": [ 79.9320793151855, 343.463024139404, 0.760430693626404, 87.3958969116211, 334.506443023682, 0.80179488658905, 72.46826171875, 333.013679504395, 0.811470806598663, 99.338005065918, 338.984733581543, 0.828160524368286, 53.0623359680176, 334.506443023682, 0.82732355594635, 117.251167297363, 388.245929718018, 0.792685210704803, 30.6708831787109, 382.274875640869, 0.776597559452057, 111.280113220215, 456.913051605225, 0.784640848636627, 11.2649574279785, 449.449234008789, 0.736559152603149, 72.46826171875, 404.666328430176, 0.806424021720886, 6.78666687011719, 506.174247741699, 0.650886654853821, 81.4248428344727, 518.116355895996, 0.690829515457153, 35.1491737365723, 515.130828857422, 0.713027656078339, 76.9465522766113, 625.595329284668, 0.824519991874695, 30.6708831787109, 619.62427520752, 0.814788639545441, 72.46826171875, 721.132194519043, 0.812629520893097, 29.1781196594238, 707.697322845459, 0.802818536758423, 76.9465522766113, 291.216300964355, 0.772570133209229, 73.9610252380371, 365.854476928711, 0.882087588310242, 59.033390045166, 515.130828857422, 0.980525076389313, 50.0768089294434, 750.987464904785, 0.787731468677521, 14.2504844665527, 739.045356750488, 0.781021475791931, 66.4972076416016, 753.972991943359, 0.787867844104767, 3.80113983154297, 734.567066192627, 0.780524611473083, 72.46826171875, 736.059829711914, 0.763829290866852, 35.1491737365723, 725.610485076904, 0.779185891151428 ], "pose_keypoints_3d": [ ] } ], "version": 1.3 } ```
DeepLabCut single animal format (one file for all frames) ``` csv scorer,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000,DLC_resnet50_skelePoseJan15shuffle1_500000 bodyparts,eye_l,eye_l,eye_l,eye_r,eye_r,eye_r,ear_l,ear_l,ear_l,ear_r,ear_r,ear_r,nose,nose,nose coords,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood 0,1788.197754,582.8817749,0.893398941,1778.797485,590.4447632,0.127174705,1805.251709,564.6546631,0.910144508,1784.953125,570.2262573,0.107157677,1790.747925,594.1011353,0.935636997 1,1788.222168,582.8106689,0.894531071,1778.798584,590.4085083,0.126578912,1805.238403,564.5864868,0.909093618,1784.94397,570.2024536,0.104491554,1790.743652,594.0860596,0.935546935 2,1788.238525,582.7897339,0.895584881,1778.815552,590.4252319,0.130826131,1805.242188,564.5612793,0.908809125,1784.940308,570.2030029,0.10585627,1790.789795,594.0892334,0.93606174 3,1788.232666,582.7907104,0.893335998,1778.824585,590.4157104,0.128489673,1805.268555,564.6519775,0.910064697,1784.994995,570.2485962,0.106905445,1790.754761,594.0883179,0.935919583 4,1788.237061,582.7630005,0.894391239,1778.817749,590.4083862,0.127810478,1805.283081,564.617981,0.909866989,1784.976074,570.2377319,0.106576219,1790.801514,594.0666504,0.935323417 5,1788.181152,582.6789551,0.895669937,1778.8573,590.4202881,0.132273525,1805.337524,564.6123657,0.910025418,1784.921753,570.1697998,0.110722385,1790.775391,594.0305176,0.936482549 6,1788.338623,582.8166504,0.893769264,1778.895508,590.444519,0.129744262,1805.33606,564.6570435,0.909310341,1785.017456,570.2346191,0.10804145,1790.679932,594.0999146,0.935638368 7,1788.342651,582.8238525,0.892479539,1778.896851,590.4647827,0.12789692,1805.365234,564.6409302,0.908787131,1785.026855,570.2400513,0.107544251,1790.679077,594.1221924,0.935059667 8,1788.348633,582.8262939,0.893451214,1778.903931,590.4279785,0.129449219,1805.356079,564.6516113,0.909340203,1785.007568,570.2471313,0.108491346,1790.665039,594.0901489,0.935388207 9,1788.395996,582.7037964,0.89509505,1778.844849,590.4187012,0.131914169,1805.461548,564.6132202,0.905206919,1784.903687,570.2348633,0.105487868,1790.762085,594.0391235,0.933868289 10,1788.44104,583.1160889,0.893697739,1779.070313,590.7844849,0.132791132,1805.308594,564.6127319,0.912624896,1784.978149,570.3155518,0.117106639,1790.90686,594.2011719,0.936214805 11,1788.567383,583.0030518,0.894327939,1778.897583,590.8908081,0.127592504,1805.494873,564.6373291,0.912322581,1784.929565,570.5378418,0.115896821,1790.853882,594.3178101,0.932211339 12,1788.585449,583.0975342,0.892759144,1779.061279,590.8235474,0.12571305,1805.421875,564.7701416,0.911573291,1784.955566,570.451416,0.114519648,1790.945801,594.2923584,0.933272541 13,1788.643066,583.1628418,0.889103591,1778.974976,591.0488281,0.125203311,1805.569336,564.8320923,0.913689613,1784.978271,570.6643677,0.113119982,1790.993774,594.3839111,0.930438876 ```
DeepLabCut multi animal format (one file for all frames) ``` csv scorer,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000,DLC_dlcrnetms5_BMX_MonoApr21shuffle1_110000 individuals,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX,BMX bodyparts,HandleL,HandleL,HandleL,HandleR,HandleR,HandleR,Fork,Fork,Fork,FrameFront,FrameFront,FrameFront,FrameUp,FrameUp,FrameUp,FrameDown,FrameDown,FrameDown,FrameBack,FrameBack,FrameBack,PedalL,PedalL,PedalL,PedalR,PedalR,PedalR coords,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood,x,y,likelihood 0,553.72,328.333,1.0,,,0.0,651.958,508.183,0.3449999988079071,701.98,461.062,0.035999998450279236,701.159,470.962,1.0,690.518,524.853,1.0,729.276,563.818,1.0,658.889,540.449,1.0,719.511,511.489,1.0 1,554.328,327.485,1.0,,,0.0,653.75,508.197,0.9869999885559082,667.948,429.661,0.0689999982714653,701.362,468.618,1.0,690.784,523.411,1.0,729.793,563.314,1.0,658.546,541.115,1.0,720.002,509.932,1.0 2,554.646,327.802,1.0,764.64,356.395,0.03200000151991844,652.77,508.075,0.9929999709129333,667.315,431.523,0.7820000052452087,700.833,468.113,1.0,690.37,522.945,1.0,730.089,563.272,1.0,658.758,539.676,1.0,720.944,511.181,1.0 3,555.489,327.213,1.0,764.234,356.033,0.1850000023841858,646.72,507.848,1.0,666.371,428.494,1.0,700.437,467.868,1.0,691.208,523.881,1.0,730.35,562.91,1.0,657.448,540.037,1.0,721.005,510.766,1.0 4,554.946,327.225,1.0,764.528,348.244,0.04100000113248825,652.795,507.692,1.0,666.354,429.879,0.9990000128746033,702.415,468.748,1.0,690.523,523.405,1.0,729.796,562.929,1.0,656.751,539.579,1.0,720.681,512.018,1.0 5,555.99,326.791,1.0,763.953,348.227,0.26100000739097595,645.027,508.144,0.8420000076293945,661.937,428.431,1.0,701.697,467.722,1.0,691.151,523.389,1.0,729.798,562.534,1.0,657.454,537.514,1.0,720.563,510.226,1.0 6,556.325,326.987,1.0,763.094,347.501,0.6840000152587891,647.477,507.464,1.0,669.293,420.073,0.9940000176429749,701.622,467.153,1.0,690.16,523.919,1.0,728.521,563.663,1.0,657.695,538.057,1.0,721.492,512.623,1.0 ```
MMPose format (one file for all frames). I don't have any file handy, but here is the structure (see there) ```code result = { 'visualization': [ # number of elements: batch_size (defaults to 1) vis_image_1, ... ], 'predictions': [ # pose estimation result of each image # number of elements: batch_size (defaults to 1) [ # pose information of each detected instance # number of elements: number of detected instances {'keypoints': ..., # instance 1 'keypoint_scores': ..., ... }, {'keypoints': ..., # instance 2 'keypoint_scores': ..., ... }, ] ... ] } ```
mprib commented 14 hours ago

@davidpagnon,

This is incredibly helpful to have as a reference, thank you! It's great to see how others approach the problem of storing this data. An ecosystem of open source tools that can interoperate to facilitate mocap is near and dear to my heart, so this is much appreciated.

Mac