Question on maintaining color and intensity across texrecon invocations

[vnarasimhan]

Hi,

I have setup a camera rig with say 50 cameras in a spherical fashion pointing towards the origin. They are wired to behave as a global shutter thru a trigger system. Lets say that the rig can do 30 FPS.

For every trigger pulse, a XFrame is created. One XFrame contains 50 images. I have 30 such XFrames for one second.

I have a moving human subject at the origin and I use a MVS pipeline (say COLMAP) to generate the point cloud for each XFrame.

After this, I am using TexRecon to generate the textures for each XFrame.

When I animate thru the Texture Mapped XFrames, I notice that the color / brightness varies between frames.

I have tried disabling seam leveling, but this introduces noticeable artifacts.

Is this problem solvable in the current implementation of Texrecon? Is there a way to average the color and gamma across frames as a part of the Texrecon invocation itself, that is, carry over information from the previous frame to the current frame? If so, where should I be looking?

Thanks.

[nmoehrle]

I believe that on such a controlled case you will get the best results by disabeling the global but keeping the local seam leveling. Have you tried that combination?

[vnarasimhan]

Our observation is that enabling both local and global seam leveling offers the best quality.

At a glance, the attachments exhibit only minimal differences and may not appear to back the statements I am making. In fact, enabling both seams appears to make the image bad in the vicinity of the abdomen. However, we have tested texrecon on more data sets and we observe pronounced patches / blotches and deteriorating quality going from all seams to local only to no seams.

[vnarasimhan]

I am posting three frames from a small sequence. Texrecon with both global as well as local seam leveling was used. One can observe the difference in colors cropping up between the frames. Look at the T-shirt.

The command line looks like so: /app/mvs-texturing/build/apps/texrecon/texrecon --no_intermediate_results /app/result/5e62c291-v/frame-0/sparse.nvm /app/result/5e62c291-v/frame-0/merge4_78_smooth_clean.ply /app/result/5e62c291-v/frame-0/allSeams-0

Below is a tiled video created from the output of a few specific cams in the rig. Shows what the real T-shirt looks like. We do not have flickering lights in the rig.

https://streamable.com/s/egesx/gwsicq

[nmoehrle]

To me it appears as if the seam leveling is trying to hide misprojections due to an inaccurate reconstruction and then blending things that do not belong together. We had issues before with inaccurate reconstructions that the gradient magnitude data term favors background foreground transitions. I think you do not have the issue of out of focus blur and could thus avoid those artifacts by using the area data term. This could help to get better results especially for the non-reflective materials where the color differences that I see here should not appear.

[vnarasimhan]

Trying out the comment from nmoehrle above, have attached a few sample frames from the same sequence. Command line looks like this: /app/mvs-texturing/build/apps/texrecon/texrecon --no_intermediate_results -d area /app/result/5e62c291-v/frame-28/sparse.nvm /app/result/5e62c291-v/frame-28/merge4_78_smooth_clean.ply /app/result/5e62c291-v/frame-28/allSeams-28

Definitely better results than seam leveling (my opinion).

[nmoehrle]

That looks a lot better :-) Improving the geometry further will improve the texture as well. Hope this works for you now :-)