SimonGiebenhain
commented
1 month ago Hi elimarinoiu,
I did some experiments with a normal detector with FFHQ images, but decided not to include it in the repo since I wasn't sure about the benefits. FYI you can look at this repo for the normal detector that I used.
I never used normals for tracking in videos. It is hard to tell what causes the complete failure that you are experiencing, but here a few things that you can look at:
- How did metrical-tracker perform on the video? Could the head pose be tracked reliably?
- Are there extreme head poses or facial expressions in the sequence? If yes does the tracking work on a subset of the sequence which excludes extreme angles/expressions?
- Are the detected landmarks, forground masks and segmentation masks looking good? (if there is a face in the background this can cause issues, but also extreme angles)
- Strong shadows are also not ideal
- Does it work to extract the first video from the video and reconstruct that single image?
- are you using
--no-intrinsics_provided?
Kind regards, Simon
elimarinoiu
Hi,
I tried to run the code on a video sequence of my own and the results look deformed. I noticed that after running the preprocessing code there was no normals directory generated - as I have seen in the provided FFHQ examples for which the 3d reconstructions look very good. I have also noticed that face_normals/pretrained_models/ is empty. This directory is created when installing the associated code in running install_preprocessing_pipelines.sh but I have failed to see where this code is called - I would have expected in preprocessing/run.sh.
My questions are 1) am I missing something regarding the computing of the normals? Do I need to run something else and/or download the associated model? 2) Do you think the missing precomputed normals could cause the deformed results on my data?
Thanks!