MANO fitting error

[ZhengdiYu]

Hi,

I have tested your MANO parameter across the validation datasets and found some of them have relatively large fitting error, is this correct?

**80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303070/380125 [21:24<07:54, 162.54it/s]0_ROM09_Interaction_Fingers_Touching_400362_29700, left Fitting error of the second method: 41.83678922283781 mm 0_ROM09_Interaction_Fingers_Touching_400363_29700, left Fitting error of the second method: 41.860034555889726 mm 0_ROM09_Interaction_Fingers_Touching_400364_29700, left Fitting error of the second method: 41.83920151295232 mm 0_ROM09_Interaction_Fingers_Touching_400365_29700, left Fitting error of the second method: 41.819588130377 mm 0_ROM09_Interaction_Fingers_Touching_400366_29700, left Fitting error of the second method: 41.869082708114455 mm 0_ROM09_Interaction_Fingers_Touching_400367_29700, left Fitting error of the second method: 41.83719895441215 mm 0_ROM09_Interaction_Fingers_Touching_400368_29700, left Fitting error of the second method: 41.86414651400453 mm 0_ROM09_Interaction_Fingers_Touching_400369_29700, left Fitting error of the second method: 41.80935444574154 mm 0_ROM09_Interaction_Fingers_Touching_400371_29700, left Fitting error of the second method: 41.819072224520326 mm 0_ROM09_Interaction_Fingers_Touching_400372_29700, left Fitting error of the second method: 41.852287764555136 mm 0_ROM09_Interaction_Fingers_Touching_400374_29700, left Fitting error of the second method: 41.79168010361514 mm 0_ROM09_Interaction_Fingers_Touching_400375_29700, left Fitting error of the second method: 41.831668630049705 mm 0_ROM09_Interaction_Fingers_Touching_400377_29700, left Fitting error of the second method: 41.838893810136675 mm 0_ROM09_Interaction_Fingers_Touching_400378_29700, left Fitting error of the second method: 41.834763321877546 mm 0_ROM09_Interaction_Fingers_Touching_400379_29700, left Fitting error of the second method: 41.82786017949213 mm 0_ROM09_Interaction_Fingers_Touching_400380_29700, left Fitting error of the second method: 41.82002676431866 mm 0_ROM09_Interaction_Fingers_Touching_400399_29700, left Fitting error of the second method: 41.8202917372794 mm 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303087/380125 [21:24<07:47, 164.66it/s]0_ROM09_Interaction_Fingers_Touching_400400_29700, left Fitting error of the second method: 41.83104801906732 mm 0_ROM09_Interaction_Fingers_Touching_400401_29700, left Fitting error of the second method: 41.82369010800102 mm 0_ROM09_Interaction_Fingers_Touching_400404_29700, left Fitting error of the second method: 41.81175568172949 mm 0_ROM09_Interaction_Fingers_Touching_400405_29700, left Fitting error of the second method: 41.81247005217557 mm 0_ROM09_Interaction_Fingers_Touching_400406_29700, left Fitting error of the second method: 41.814299689270776 mm 0_ROM09_Interaction_Fingers_Touching_400407_29700, left Fitting error of the second method: 41.82162838330069 mm 0_ROM09_Interaction_Fingers_Touching_400408_29700, left Fitting error of the second method: 41.79851802312422 mm 0_ROM09_Interaction_Fingers_Touching_400410_29700, left Fitting error of the second method: 41.8551337193332 mm 0_ROM09_Interaction_Fingers_Touching_400411_29700, left Fitting error of the second method: 41.86795455212615 mm 0_ROM09_Interaction_Fingers_Touching_400415_29700, left Fitting error of the second method: 41.86184075271003 mm 0_ROM09_Interaction_Fingers_Touching_400416_29700, left Fitting error of the second method: 41.85177738010605 mm 0_ROM09_Interaction_Fingers_Touching_400417_29700, left Fitting error of the second method: 41.807482953442694 mm 0_ROM09_Interaction_Fingers_Touching_400418_29700, left Fitting error of the second method: 41.89110493080544 mm 0_ROM09_Interaction_Fingers_Touching_400419_29700, left Fitting error of the second method: 41.819900822177885 mm 0_ROM09_Interaction_Fingers_Touching_400421_29700, left Fitting error of the second method: 41.81178927798937 mm 0_ROM09_Interaction_Fingers_Touching_400422_29700, left Fitting error of the second method: 41.79978573291985 mm 0_ROM09_Interaction_Fingers_Touching_400425_29700, left Fitting error of the second method: 41.85476559573429 mm 0_ROM09_Interaction_Fingers_Touching_400426_29700, left Fitting error of the second method: 41.893154244733786 mm 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303105/380125 [21:24<07:41, 166.79it/s]0_ROM09_Interaction_Fingers_Touching_400428_29700, left Fitting error of the second method: 41.82420296166631 mm 0_ROM09_Interaction_Fingers_Touching_400430_29700, left Fitting error of the second method: 41.89487546307009 mm 0_ROM09_Interaction_Fingers_Touching_400431_29700, left Fitting error of the second method: 41.87506773995345 mm 0_ROM09_Interaction_Fingers_Touching_400432_29700, left Fitting error of the second method: 41.85703023101172 mm 0_ROM09_Interaction_Fingers_Touching_400433_29700, left Fitting error of the second method: 41.80758921351955 mm 0_ROM09_Interaction_Fingers_Touching_400435_29700, left Fitting error of the second method: 41.89182090205234 mm 0_ROM09_Interaction_Fingers_Touching_400436_29700, left Fitting error of the second method: 41.830865113544654 mm 0_ROM09_Interaction_Fingers_Touching_400437_29700, left Fitting error of the second method: 41.83728497799924 mm 0_ROM09_Interaction_Fingers_Touching_400439_29700, left Fitting error of the second method: 41.84445448179384 mm 0_ROM09_Interaction_Fingers_Touching_400440_29700, left Fitting error of the second method: 41.8601474769388 mm 0_ROM09_Interaction_Fingers_Touching_400441_29700, left Fitting error of the second method: 41.84431378484727 mm 0_ROM09_Interaction_Fingers_Touching_400442_29700, left Fitting error of the second method: 41.81398618138446 mm 0_ROM09_Interaction_Fingers_Touching_400447_29700, left Fitting error of the second method: 41.83456354366176 mm 0_ROM09_Interaction_Fingers_Touching_400448_29700, left Fitting error of the second method: 41.83495966100603 mm 0_ROM09_Interaction_Fingers_Touching_400449_29700, left Fitting error of the second method: 41.83634226214207 mm 0_ROM09_Interaction_Fingers_Touching_400451_29700, left Fitting error of the second method: 41.86285194894558 mm 0_ROM09_Interaction_Fingers_Touching_400453_29700, left Fitting error of the second method: 41.90061512827506 mm 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303122/380125 [21:24<07:42, 166.53it/s]0_ROM09_Interaction_Fingers_Touching_400456_29700, left Fitting error of the second method: 41.85104057425875 mm 0_ROM09_Interaction_Fingers_Touching_400460_29700, left Fitting error of the second method: 41.82334442221328 mm 0_ROM09_Interaction_Fingers_Touching_400464_29700, left Fitting error of the second method: 41.85632879936446 mm 0_ROM09_Interaction_Fingers_Touching_400469_29700, left Fitting error of the second method: 41.83015675501236 mm 0_ROM09_Interaction_Fingers_Touching_400475_29700, left Fitting error of the second method: 41.867777910153805 mm 0_ROM09_Interaction_Fingers_Touching_400479_29700, left Fitting error of the second method: 41.90044827469683 mm 0_ROM09_Interaction_Fingers_Touching_400480_29700, left Fitting error of the second method: 41.86874301363451 mm 0_ROM09_Interaction_Fingers_Touching_400481_29700, left Fitting error of the second method: 41.85659026174667 mm 0_ROM09_Interaction_Fingers_Touching_400482_29700, left Fitting error of the second method: 41.88802721010183 mm 0_ROM09_Interaction_Fingers_Touching_400483_29700, left Fitting error of the second method: 41.84713555259342 mm 0_ROM09_Interaction_Fingers_Touching_400484_29700, left Fitting error of the second method: 41.80662625556496 mm 0_ROM09_Interaction_Fingers_Touching_400485_29700, left Fitting error of the second method: 41.8349112306501 mm 0_ROM09_Interaction_Fingers_Touching_400486_29700, left Fitting error of the second method: 41.89315674538595 mm 0_ROM09_Interaction_Fingers_Touching_400487_29700, left Fitting error of the second method: 41.87659436663791 mm 0_ROM09_Interaction_Fingers_Touching_400488_29700, left Fitting error of the second method: 41.81879467896671 mm 0_ROM09_Interaction_Fingers_Touching_400489_29700, left Fitting error of the second method: 41.838183801803766 mm 0_ROM09_Interaction_Fingers_Touching_400500_29700, left Fitting error of the second method: 41.85092418763113 mm 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303139/380125 [21:24<07:41, 166.97it/s]0_ROM09_Interaction_Fingers_Touching_400501_29700, left Fitting error of the second method: 41.83705548424705 mm 0_ROM09_Interaction_Fingers_Touching_400502_29700, left Fitting error of the second method: 41.86292053793695 mm 0_ROM09_Interaction_Fingers_Touching_400503_29700, left Fitting error of the second method: 41.86674843982388 mm**

These are some of them that have fitting error larger than 20mm.

[mks0601]

Please check two things 1) Did you check 3D joint coordinates from the annotation and MANO are in the same coordinate system? 2) Did you check the joint_valid, which indicates whether joints annotations are valid or not

[ZhengdiYu]

Please check two things

1. Did you check 3D joint coordinates from the annotation and MANO are in the same coordinate system?
2. Did you check the joint_valid, which indicates whether joints annotations are valid or not

Thanks for your reply. I just used your convert.py. I also used your render.py to get a result for this frame from cam400361:

it looks reasonable actually. But the fitting error is: Right: Fitting error of the second method: 5.244508882705751 mm Left: itting error of the second method: 41.813552967832784 mm

[ZhengdiYu]

@mks0601 I just now double checked using your render.py and first projection method, it looks literally the same as the image I just posted, including the picture and fitting error: right: 5.022617mm, left: 41.82419mm

[ZhengdiYu]

@mks0601 Any updates about this ? Is this correct?

[mks0601]

What do you mean by the first projection method?

[ZhengdiYu]

What do you mean by the first projection method?

In your convert.py, there are two projection method: one directly applies extrinsic to mesh and joints while another one applies to MANO parameter. Both of them gave the same results

[mks0601]

I guess you have old version of render.py? Current one does not have that.

[ZhengdiYu]

I guess you have old version of render.py? Current one does not have that.

No I mean that I have used both projection method in convert.py, I think your convert.py have two methods in current version, and both two methods gave me similar fitting error.

And, secondly, I have tried your original render.py which only use the first projection method in conver.py, and I also tried to replace the method to projection method 2, which also gave me same projection results.

I've already confirmed this in two machine, with different environment, it seems that it's indeed that large. But only happens to several mano parameters and thousands of frames in validation set.

In other split, there are also some mano parameters with larger fitting error, I didn't count how many there are.

[mks0601]

Could you let me know split, capture_idx, seq_name, and cam_idx? I do not face such large errors of the left hand in random sequences.

[ZhengdiYu]

Could you let me know split, capture_idx, seq_name, and cam_idx? I do not face such large errors of the left hand in random sequences.

You can just randomly pick one from above, for example:

0_ROM09_Interaction_Fingers_Touching_400363_29700, Capture 0, Seq:ROM09_Interaction_Fingers_Touching, cam: 400363, frame 29700 left Fitting error of the second method: 41.860034555889726 mm

I think it's frame related, some frame will have large fitting error across all the camera_idx.

[mks0601]

Please give me exact split, capture_idx, seq_name, cam_idx to reproduce your results

[ZhengdiYu]

Please give me exact split, capture_idx, seq_name, cam_idx to reproduce your results

split = 'val' capture_idx = '0' seq_name = 'ROM01_No_Interaction_2_Hand' cam_idx = '400361' frame_idx = '29700'

[mks0601]

I found that val/Capture0/ROM01_No_Interaction_2_Hand/cam400361 does not have image29700.jpg Please check your image folder.

[ZhengdiYu]

ROM09_Interaction_Fingers_Touching

Sorry the seq_name is ROM09_Interaction_Fingers_Touching. I didn't change seq_name in my script because it's not related to the fitting error and the problem.

[ZhengdiYu]

I found that val/Capture0/ROM01_No_Interaction_2_Hand/cam400361 does not have image29700.jpg Please check your image folder.

Have you got the results for /Capture0/ROM09_Interaction_Fingers_Touching/cam400361/image29700.jpg ? I just double-checked and I think you might get results similar to mine: 41.813552967832784 mm for left and 5.244508882705751 mm for right

[mks0601]

[mks0601]

[mks0601]

This is because 3D GTs are not correct for that frame.

[mks0601]

As there are milions of samples in our datasets, some weird 3D GTs marginally affect the error averaged across all samples. This is inherent failure case of marker-less MoCap datasets

[ZhengdiYu]

As there are milions of samples in our datasets, some weird 3D GTs marginally affect the error averaged across all samples. This is inherent failure case of marker-less MoCap datasets

Thanks for your help! I think I'll have to check all of them and drop these points.

[ZhengdiYu]

Sorry for reopen this issue. I just want to confirm that did you use the wrong 3D GTs for evaluation? Is there a way to eliminate these? I think the joint_valid could not eliminate these.

[mks0601]

Actually, there is no perfect way to check the obtained GTs are correct or not. We checked that our obtained annotations have 2.78 mm error (reported in the paper) on our human-annotated test set. As I said above, most of samples have correct (or near correct) 3D GTs, so the average error would be marginally affected by few wrong GTs.

[ZhengdiYu]

marginally

OK. Thanks!