search

radekd91 / inferno

🔥🔥🔥 Set the world of 3D faces on fire with INFERNO 🔥🔥🔥
Other
170 stars 18 forks source link

Looking forward to the release of FaceReconstruction model #2

Open YifengMa9 opened 10 months ago

YifengMa9 commented 10 months ago

Marvelous project! Looking forward to the release of FaceReconstruction model.

wzx7084 commented 10 months ago

+1

radekd91 commented 10 months ago

Coming very soon. Just have to put together a little demo script :-). Stay tuned!

radekd91 commented 9 months ago

Face reconstruction model is out. The demos are available. Please take a look. :-)

RainEggplant commented 9 months ago

Face reconstruction model is out. The demos are available. Please take a look. :-)

Thank you for this great work! But the checkpoint of EMICA-CVT_flame2020_notexture seems to be missing from the link provided. Would you mind checking it?

radekd91 commented 8 months ago

Can you check again? I fixed a few issues, I'm hoping your problem went away.

luizgh commented 8 months ago

Hi @radekd91 - not OP, but I just tried the demo and there seems to still be some issues:

  1. The ffmpeg comman seems to be broken (https://github.com/radekd91/inferno/blob/master/inferno/datasets/FaceVideoDataModule.py#L197-L202 - need to comment line 197) - so if you don't have a cache of the detections it will break
  2. There seems to be some issue with batching vs MICA being run once, I am getting this error: RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 4 for tensor number 1 in the list.. It goes away if you set the batch size to 1: https://github.com/radekd91/inferno/blob/master/inferno_apps/FaceReconstruction/demo/demo_face_rec_on_video.py#L62

Those are some issues I ran onto so far. Even fixing them I ran into some other problems that I will investigate next week if I have time.

I appreciate all the effort on this - looking forward to seeing how well the FaceReconstruction model is working!

radekd91 commented 8 months ago

What version of ffmpeg do you have? The command is ruining fine for me

luizgh commented 8 months ago

I am using ffmpeg 5.1.2.

But notice on: https://github.com/radekd91/inferno/blob/master/inferno/datasets/FaceVideoDataModule.py#L197-L202

Line 197 is setting the "out_format" with the flags, not just the output filename - so line 202 will be using -r 1 -i <source> twice. If I comment out line 197 this part works fine.

Example (without the fix), I get this on the args: ['ffmpeg', '-r', '1', '-i', 'dev/inferno/assets/data/EMOCA_test_example_data/videos/82-25-854x480_affwild2.mp4', '-r', '1', '-r 1 -i dev/inferno/assets/data/EMOCA_test_example_data/videos/82-25-854x480_affwild2.mp4 -r 1 "video_output/processed_2024_Jan_02_07-46-06/82-25-854x480_affwild2/videos/%06d.png"']

KangZW-2001 commented 5 months ago

@luizgh hello, may I ask how to fix this error RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 4 for tensor number 1 in the list, and when you fix it, does it has influence to other part ?

luizgh commented 5 months ago

I just set batch_size=1 on https://github.com/radekd91/inferno/blob/master/inferno_apps/FaceReconstruction/demo/demo_face_rec_on_video.py#L62

radekd91 commented 5 months ago

This looks like it can't find ffmpeg. Either it's not properly installed or it's not on the path (perhaps your environment wasn't activated).

On Thu, May 30, 2024, 03:19 KangZW-2001 @.***> wrote:

And last night it worked well, I don't think I changed something except the input_video and output_folder.

— Reply to this email directly, view it on GitHub https://github.com/radekd91/inferno/issues/2#issuecomment-2139237673, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHORAHJA25IJM2ODU6DZE34L3AVCNFSM6AAAAABBB6BN6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGIZTONRXGM . You are receiving this because you were mentioned.Message ID: @.***>

KangZW-2001 commented 5 months ago

@luizgh

This looks like it can't find ffmpeg. Either it's not properly installed or it's not on the path (perhaps your environment wasn't activated). … On Thu, May 30, 2024, 03:19 KangZW-2001 @.> wrote: And last night it worked well, I don't think I changed something except the input_video and output_folder. — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHORAHJA25IJM2ODU6DZE34L3AVCNFSM6AAAAABBB6BN6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGIZTONRXGM . You are receiving this because you were mentioned.Message ID: @.>

Yeah, I got it. It is the same error happened on this part https://github.com/radekd91/inferno/blob/master/inferno/datasets/FaceVideoDataModule.py#L197-L202. It's the ffmpeg version problem.

KangZW-2001 commented 4 months ago

@radekd91 Sorry to bother you again, I think it's just a tiny question, so I just comment below this issue. I noticed that EMICA reconstruted the frame (with face) to the result, whose 'exp param' has 100 dims and 'shape param' has 300 dims. But the paper says that the 'exp param' is 50 dims. I remember the official FLAME2020 model has 50 dims 'exp params' (Maybe I'm remembering it wrong). Did you fine-tune the FLAME2020 model to adapt the more dims params ?