search

barisgecer / OSTeC

TF implementation of our CVPR 2021 paper: OSTeC: One-Shot Texture Completion
https://openaccess.thecvf.com/content/CVPR2021/html/Gecer_OSTeC_One-Shot_Texture_Completion_CVPR_2021_paper.html
Other
188 stars 28 forks source link

Render 4-channel image? #18

Closed laolongboy closed 2 years ago

laolongboy commented 2 years ago

Hi, the shape of im_masked is (3, 1120, 1120), and then concats with the 2d face segmentation mask mask_landmarks. What does the renderer do to the four-channel texture map? It will render RGB and mask channel seperately? What the new mask img_uv_src.pixels[3]<0.5 mean?

https://github.com/barisgecer/OSTeC/blob/ebf348e92833c54e2486799c2ec8efc6cf8a14a8/core/operator.py#L288-L293

Thanks.

barisgecer commented 2 years ago

Hi, A very good question. render_uv_image function basically unwrap the face into a UV space given a 3D face reconstruction (i.e. dense landmarks). In order to know which parts of UV texture map need completion, we first run a face segmentation/masking on 2D image and then map it to the UV texture map. To avoid running UV extraction (i.e. which is arasterization process) twice, I concat the masking to the RGB image and then seperate.

img_uv_src.pixels[3]<0.5 inversing the map as well as removing any non-binary outputs caused by the interpolation.

Later we use this UV mask here to remove segmented part of the face from UV texture map.

Hope this answers your question, Baris

laolongboy commented 2 years ago

Thanks. What a marvellous idea!