search

rotemtzaban / STIT

MIT License
1.2k stars 170 forks source link

Aligning images #10

Closed Carlyx closed 2 years ago

Carlyx commented 2 years ago

Hi！Your work is amazing!

But there is a problem with image alignment. (https://github.com/rotemtzaban/STIT/blob/ef851d3f0ecc1839cbd307fdfdc7fa6f981f6228/train.py#L81)

The next two rows are the input and output respectively. And the result in the second row does not look well aligned（The angle of the face posture is different）. Do you have any advice for me? 164

rotemtzaban commented 2 years ago

@Carlyx We smooth the alignment parameters as otherwise the videos end up jittery. This does compromise alignment for temporal coherency, but we haven't seen side effects of this on our videos. Do you see any problems with the end results? e.g the inversions or the edits? The alignment itself is only a stage in the pipeline, so it doesn't need to be perfect, just consistent with the rest of it.

rotemtzaban commented 2 years ago

If you want to try to remove/reduce the smoothing, you can try to play with the parameters --xy_sigma --center_sigma. Defaults are 3.0 and 1.0 respectively. To remove them completely set them to 0.0.

By the way, what is the fps of your videos? If it's lower then 24 it might be better to reduce the smoothing slightly.

Carlyx commented 2 years ago

Thank you for your reply! Actually I am just focusing on the alignment module and haven't tried the the whole pipeline. I select 4 frames from the video (10s 30fps) with large differences in head pose for testing. And when I set xy_sigma and center_sigma to 0.0, the results look much better.