search

TadasBaltrusaitis / OpenFace

OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
Other
6.72k stars 1.82k forks source link

Repeating Frame Bug #972

Closed Moktar13 closed 3 years ago

Moktar13 commented 3 years ago

Hi,

I'm having an issue with the FeatureExtraction tool and the output data that it generates - more specifically face isolation. The main issue is that the output data under the ".au_aligned" folder and the respective output video (constructed by FeatureExtraction tool), seem to have some repeating frame issue where every X frame is replaced by another frame.

Blank frame (subject is out of view) being replaced by frame where subject is in view. p1

Regular frame being replaced by irregular frame (frame bug highlighted in orange) p2

Video showing the issue demonstrated through the .avi video that FeatureExtraction creates. Link to video

Any help is much appreciated!

Desktop

OS: Ubuntu Version: 20.04

TadasBaltrusaitis commented 3 years ago

Oh wow, this looks super strange, and I haven't seen anything like this before. I wonder if the issue is with the encoded video. Any chance you could re-encode the input video (e.g. using ffmpeg)?

If you just try opening that video with OpenCV does the same behavior persist? I'm suspecting this is less an OpenFace but rather an OpenCV issue.

Moktar13 commented 3 years ago

Oh wow, this looks super strange, and I haven't seen anything like this before. I wonder if the issue is with the encoded video. Any chance you could re-encode the input video (e.g. using ffmpeg)?

I don't use the input video directly with OpenFace, I break the input video up into a series of images (frames) and then use them with FeatureExtraction. So I am unsure if changing the encoding of the input video will do anything. Sorry, I should have mentioned this in the original post!

TadasBaltrusaitis commented 3 years ago

Maybe it is some image sorting issue then. Are all images pre-pended with 000 or something similar? This actually feels like the code is loading images 0 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 etc.

Moktar13 commented 3 years ago

The input images aren't pre-pended so they're named incrementally like 1, 2, ...,100, 101, and so on. The output images (face isolation) are pre-pended first with 000001, 000002, 000010, 000011, 000120, 001500, 01250, etc. So it looks like the output images are pre-pended with five 0's, but then it decreases as the image number goes up.

TadasBaltrusaitis commented 3 years ago

In order for OpenFace to know which images to load first you will need to prepend 0's to your names. as currently it uses a sort function internally to sort names, leading to "wrong" sorting of input.

Moktar13 commented 3 years ago

I pre-pended 0's as you recommended and it seems to have fixed the issue! I will continue with more testing just to confirm this and then mark it as solved. Thanks so much for the help!