how to simultaneously use both render_silhouettes and render_rgb while training?

[yuedajiong]

how to simultaneously use both render_silhouettes and render_rgb while training?

[JiejiangWu]

Hi, Our render_rgb function in code actually renders the surface normal to an RGB image, as shown in the paper. So, if you need to use silhouettes and RGB images (not surface normal images) for supervision in training, you may need to implement the shading process (i.e., a lambert reflection).

[yuedajiong]

from my understanding, the shading is just to improve the performace. I mean: if we want to reconstruct a colored object, normally, solution#1:we need reconstruct shape by silhouette firstly, and then reconstuct the texture by rgb/a rendering. solution#2: directly reconstruct and render by render_rgb. so, the reconsuction function/network should be: inside/outside-flag, rgba-value = f_reconstruction(x, y). and call later render_rgb.
when I try to directly use render_rgb，I can not train the netowrk to convergently.

[JiejiangWu]

Your understanding is right. The shading of color provides more information, which can improve the performance. But note that, in our paper, the network does not model the rgba-value, and the render_rgb function in our code actually does the thing: render the surface normal of implicit shape. https://github.com/JiejiangWu/DFR/blob/be9603ad1f95847b571e2e06cd63451f9b5fb382/dfr/render/dfrRender.py#L543

So, if you want to render the REAL colored image (which to compare with the RGB training image for loss), you need to implement the color process (which I called the shading process in the last reply).

[yuedajiong]

Thanks, got it.

I am in China, may I contact you in WeChat? My wechat is: john122333.

I am trying to design an unified vision reconstuction algorihtm, like this:

explicit_or_implict 3d_structure_and_motion = f_unified_differentable_reconsuct(a_few_images_without_camera_pose) rgb/a, silhouette_01 = f_unified_differentable_render(explicit_or_implict 3d, timestep, camera_view, ligth, resolution, ..)

1)easy to use without camera_pose (refer-to. nope-near, colmap, ...) 2)explicit_or_implict. explicit: obj/static shape and texture, flx/amination, ... implict: any continuous function, CR-curve-for-hair, .... 3)with timestep, to generate dymamic object

yes, I finished my first version, and still on tuning.