search

gulvarol / surreal

Learning from Synthetic Humans, CVPR 2017
http://www.di.ens.fr/willow/research/surreal
Other
588 stars 106 forks source link

Material segments vertices loaded from pickle file may contains gaps #7

Open wildermuthn opened 6 years ago

wildermuthn commented 6 years ago

Thanks for making this available! I'm an amateur at both ML and Blender, but your project is fascinating, so I'm digging into it from top to bottom. I've begun generating synthetic data according to your scripts and instructions, and I noticed this:

image

This is the raw synthetic data, with Material 1 (head) vertices selected. The back is worse:

image

Update: From what I can tell, there are gaps in the faces between spine2 and head, and between hips and left/rightUpperLeg. This seems to be because the first sorted key in the list of parts doesn't actually have a segment created from the material, but whatever is left over among unassigned vertices is applied to the first material group. So although it seems that the head is being properly assigned, it actually is given anything that is left over.

I'm not sure this actually makes a difference once training is done on the synthetic data, as it seems that you join together various discrete segments into larger segments. So perhaps the gaps disappear. However, when using your pre-trained models to segment images, I do recall occasionally seeing odd parts near the hips and upper body that were mis-segmented as the head, so I think this might be a real bug. I am running Blender 2.79, so if you're not seeing the same thing, perhaps that's the culprit.

Thanks again for sharing this!

## My hack added to main_part_1.py - create_segmentation(). It seems like there ought to be a way to configure Blender to just assign these automatically. 

vsegm['rightUpLeg'].extend([4353, 4418])
vsegm['leftUpLeg'].extend([869, 932])
vsegm['spine2'].extend([709, 712, 734, 1236, 1535, 1840, 1847, 1899, 2903, 2938, 2940, 2949, 4195, 4200, 4222, 4719, 5006, 5301, 5308, 5360, 6362, 6397, 6399, 6408])

image

gulvarol commented 6 years ago

Thanks for reporting this issue. Yes the released data has such holes and the released models are trained on that data. Therefore, the models produce these gaps sometimes.

Are you visualizing a blender file produced by the script here? The pickle file (segm_per_v_overlap.pkl) that stores the segment assignments is correct. I don't understand why Blender does not assign the proper faces to the head. If the hack does the job, it can be a temporary workaround. I haven't ran it myself yet.