FaceLandmarkVid 1000+ fps issue

[imants-meidrops]

I set up OpenFace on my local. The FaceLandmarkImg works fine and produces an output. But when using Vid (both webcam and sample video files) the screen loads up and buffers insanely showing a frame every couple of seconds, it shows as having astronomical FPS values.

I am using Mac M1.

[brmarkus]

Using which option from "https://github.com/TadasBaltrusaitis/OpenFace/wiki/Mac-Installation" have you built it? There were quite a few MAC-related issues reported, with solutions or workarounds found - but not recently (using Apple silicon, but Intel silicon)?

[imants-meidrops]

No build issues. As I said it works fine for Images, but not for video.

[brmarkus]

What application do you use - and how? C++, Python, C#? The application is not a "realtime video player", i.e. it is processing on a frame-by-frame base.

Landmark detection is pretty fast, i.e. the single frames will be processed very quickly one after the other... not considering timestamps within the video-file, not considering the video's framerate...

You are using a powerful machine... 1000 "fps" for only doing landmark detection is "normal".

Feel free to integrate it into a media-player framework like gstreamer... or read the video's framerate, take the time processing the current video frame takes and insert a "sleep()" ("framerate minus time occupied by processing").

[imants-meidrops]

I was using this command build/bin/FaceLandmarkVid -f pathToFile/videosample1.avi The problem is that it doesn't really work like in the demo video https://www.youtube.com/watch?v=V7rV0uy7heQ&ab_channel=Tadyla. It freezes and shows a frame every 5-10 seconds. (and shows 1000+ fps in the fps text)

[brmarkus]

Do you have a chance to double-check your OpenCV version (e.g. a small test-app)? Have you built it (as under "Option B")? Have you tried other video files? OpenCV might do video-decoding with CPU instead of the GPU (I'm not aware of M1 being supported by OpenCV for doing GPU-accelerated video decoding).

Have you tried other video files? The sample videos under "https://github.com/TadasBaltrusaitis/OpenFace/tree/master/samples" have quitre exotic video-codecs (WMV2, DIVX).

UPDATE: Which framerate gets displayed with single images?

[imants-meidrops]

I followed the guide under option A. I have tried using a webcam input as well.

I am using OpenCV version 4.6.0

The same issue persists with using sample .wmv files and webcam feed.

When feeding in a single image to FaceLandmarkVid I got 353 FPS.

[brmarkus]

Can you check with the section "https://github.com/TadasBaltrusaitis/OpenFace/wiki/Mac-Installation#openblas-performance" under Open-B (exporting ENV variables, (re-)installing OpenBLAS with OpenMP)?

OpenCV goves different paths with picture-decoding (usually directly via image-codecs) and video-frame-decoding (via gstreamer or ffmpeg and then via SW- or HW-video-codecs), i.e. the input material could influence the throughput.

Not sure about your webcam feed - is it raw, Bayer, compressed/h.264/h.265?

Can you try with other video-files, other than from the "samples" folder, e.g. "normal" MP4 files with h.264, or MPG(2)-videos?

UPDATE: Now I tried it on an Intel Laptop under MS-Win10 in a WSL2-Ubuntu-environment - using the pre-build Docker container image, as described under "https://github.com/TadasBaltrusaitis/OpenFace/tree/master/docker". For images, it just worked. For videos, I needed to start the container by adding -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -h $HOSTNAME -v $HOME/.Xauthority:/root/.Xauthority. Then I was able to see the video playback smootly. The laptop is not very powerful, I get only 26-fps...:

Your issue might be Apple-specific and/or environment-specific. Do you want to give it a try by building the container on your computer (I think the pre-built image is for X86)...

[imants-meidrops]

I will give it a try to build it using docker. Will let you know if that fixes the issue.

[brmarkus]

Maybe someone in the community can give it a try on a Mac as well for comparison?

[Kenon-mo]

Hi, I had the same issue on Manjaro Linux (no apple silicon), for me setting those env vars helped:

export OMP_NUM_THREADS=1
export VECLIB_MAXIMUM_THREADS=1