OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
Other
6.71k
stars
1.82k
forks
source link
FaceAnalyzer DetectLandmarks completely crashes PI 5 #1087
I am new to OpenFace, and am trying to get started writing a simple program that captures an image from a webcam every half a second, and output "BLINK!" when it detects a blink.
Describe the bug
(Source code on this issue below)
This was tested on two systems.
I have a for loop that constantly uses a webcam to capture a frame, converts it to grayscale with openCV using cv::cvtColor(frame, grayscale, cv::COLOR_BGR2GRAY);. Once I do that, I use face_model.DetectLandmarks(grayscale, det_params) to try and detect landmarks, so that I can look up the "codes" (I forgot what they're officially called).
When running on the raspberry pi, the program will loop through an arbitrary amount of times (from as little as 3 to as much as 20 loops of this) before the entire pi just shuts down completely. No error message, just have to unplug and plug it back in.
On my dell poweredge ubuntu server, it will loop over this forever without crashing.
It is important to note however, that on both setups, the DetectLandmarks function always returns FALSE, indicating that it was unsuccessful, but I figured that would be a bridge I crossed at a different point.
Even worse, the program always crashes between COUT lines.
Things I have ruled out:
It is not a memory issue. Memory peaks at maybe half usage whilst it is running.
It is not the CPU getting overloaded. CPU peaks at 25% usage
It is not temperature. Temperature peaks at 60-65C
It is not a memory leak. I ran valgrind, and found nothing of note
It is not a power issue. Originally I thought maybe the webcam was drawing too much power. However, if I comment out the DetectLandmarks, it will run just fine forever. I have confirmed the webcam is working correctly, and the device can handle constant images being captured.
SOURCE CODE:
#include <iostream>
#include <csignal>
#include <opencv2/opencv.hpp>
#include <FaceAnalyser.h>
#include <FaceAnalyserParameters.h>
#include <GazeEstimation.h>
#include <LandmarkCoreIncludes.h>
int DELAY = 500;
cv::VideoCapture cap;
bool keep_running = true;
// Signal handler for SIGINT
void handle_sigint(int sig) {
std::cout << "Interrupt signal (" << sig << ") received. Cleaning up..." << std::endl;
if (cap.isOpened()) {
cap.release();
}
cv::destroyAllWindows();
keep_running = false;
}
int main() {
std::signal(SIGINT, handle_sigint);
std::cout << "Starting webcam..." << std::endl;
// Open the default webcam
cap.open(0);
if (!cap.isOpened()) {
std::cerr << "Error: Could not open webcam" << std::endl;
return -1;
}
cap.set(cv::CAP_PROP_FRAME_WIDTH, 320);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, 240);
// Load face model
LandmarkDetector::FaceModelParameters det_params;
det_params.model_location = "/usr/local/etc/OpenFace/model/main_ceclm_general.txt"; // Ensure the correct model path
LandmarkDetector::CLNF face_model(det_params.model_location);
if (!face_model.loaded_successfully) {
std::cerr << "Error: Could not load face model from " << det_params.model_location << std::endl;
return -1;
}
// Initialize FaceAnalyser with FaceAnalyserParameters
FaceAnalysis::FaceAnalyserParameters params;
FaceAnalysis::FaceAnalyser face_analyser(params);
std::cout << "Libraries initialized successfully." << std::endl;
while (keep_running) {
cv::Mat frame;
cap >> frame;
if (frame.empty()) {
std::cerr << "Error: Could not capture frame" << std::endl;
break;
}
//Convert to grayscale
cv::Mat grayscale;
std::cout << "Before grayscale convert | ";
cv::cvtColor(frame, grayscale, cv::COLOR_BGR2GRAY);
std::cout << "After grayscale convert | " << std::endl;
///////////////////////////////////////
//THIS IS WHERE IT ALWAYS DIES//
///////////////////////////////////////
// Detect landmarks
std::cout << "Before detect landmarks | ";
bool detection_success;
try {
detection_success = false;
//detection_success = face_model.DetectLandmarks(grayscale, det_params);
} catch (const std::exception &e) {
std::cerr << "Exception during DetectLandmarks: " << e.what() << std::endl;
break;
} catch (...) {
std::cerr << "Unknown exception during DetectLandmarks." << std::endl;
break;
}
std::cout << "After detect landmarks" << std::endl;
if (detection_success) {
std::cout << "DETECT SUCCESS" << std::endl;
face_analyser.AddNextFrame(frame, face_model.detected_landmarks, detection_success, 0.0, false);
// Check for blinks
std::vector<std::pair<std::string, double>> aus = face_analyser.GetCurrentAUsReg();
for (const auto& au : aus) {
if (au.first == "AU45" && au.second > 0.1) {
std::cout << "BLINK!" << std::endl;
}
}
} else {
std::cout << "DETECT FAILURE" << std::endl;
}
cv::waitKey(DELAY);
std::cout << std::endl;
}
cap.release();
std::cout << "Program terminated successfully." << std::endl;
return 0;
To Reproduce
I have no idea how you would reproduce this besides cloning my project and attempting to run it. If someone would like to try to do that, let me know and I can post it.
Expected behavior
Program outputs when it detects a blink.
Screenshots
Probably not applicable
Desktop (please complete the following information):
Device specifications above
Additional context
Compiling OpenFace on my raspberry PI was a chore. I had to compile the following dependencies from source:
openCV after changing --param=ipcp-unit-growth=1000000 to --param=ipa-cp-unit-growth=1000000 in some of the CMakeLists, and then using cmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE ..
The rest of the dependencies were installed with apt-get
Then for OpenFace itself, I had to use cmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE -D OpenBLAS_INCLUDE_DIR=/usr/include/aarch64-linux-gnu -D OpenBLAS_LIB=/usr/lib/aarch64-linux-gnu/libopenblas.so ..
I also had to remove -msse flags from CMakeLists due to architecture reasons (ARM vs x86)
What I believe the problem to be
My guess is that the CPU architecture is the cause of this issue. It was already a pain enough to get this to compile on ARM, as a had to do some special steps to get it to happen (described under additional context). I've just been banging my head against this for weeks now going through stack overflow and chatGPT. Please help, I would appreciate it so much.
I am new to OpenFace, and am trying to get started writing a simple program that captures an image from a webcam every half a second, and output "BLINK!" when it detects a blink.
Describe the bug (Source code on this issue below) This was tested on two systems.
The system where the bug is happening:
The system where the bug does not happen (Dell poweredge ubuntu server)
I am running this program I created that uses the following libraries:
I have a for loop that constantly uses a webcam to capture a frame, converts it to grayscale with openCV using
cv::cvtColor(frame, grayscale, cv::COLOR_BGR2GRAY);. Once I do that, I useface_model.DetectLandmarks(grayscale, det_params)to try and detect landmarks, so that I can look up the "codes" (I forgot what they're officially called).When running on the raspberry pi, the program will loop through an arbitrary amount of times (from as little as 3 to as much as 20 loops of this) before the entire pi just shuts down completely. No error message, just have to unplug and plug it back in.
On my dell poweredge ubuntu server, it will loop over this forever without crashing.
It is important to note however, that on both setups, the
DetectLandmarksfunction always returns FALSE, indicating that it was unsuccessful, but I figured that would be a bridge I crossed at a different point.Even worse, the program always crashes between COUT lines.
Things I have ruled out:
DetectLandmarks, it will run just fine forever. I have confirmed the webcam is working correctly, and the device can handle constant images being captured.SOURCE CODE:
To Reproduce I have no idea how you would reproduce this besides cloning my project and attempting to run it. If someone would like to try to do that, let me know and I can post it.
Expected behavior Program outputs when it detects a blink.
Screenshots Probably not applicable
Desktop (please complete the following information): Device specifications above
Additional context Compiling OpenFace on my raspberry PI was a chore. I had to compile the following dependencies from source:
./configure --enable-gpl --enable-version3 --enable-nonfree --enable-small --enable-libx264 --enable-libx265 --enable-libvpx --enable-libmp3lame --enable-libopus --extra-cflags="-fPIC" --extra-ldflags="-fPIC" --disable-libfdk-aac --enable-shared --disable-static--param=ipcp-unit-growth=1000000to--param=ipa-cp-unit-growth=1000000in some of the CMakeLists, and then usingcmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE ..The rest of the dependencies were installed with apt-get
Then for OpenFace itself, I had to use
cmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE -D OpenBLAS_INCLUDE_DIR=/usr/include/aarch64-linux-gnu -D OpenBLAS_LIB=/usr/lib/aarch64-linux-gnu/libopenblas.so ..I also had to remove
-msseflags from CMakeLists due to architecture reasons (ARM vs x86)What I believe the problem to be
My guess is that the CPU architecture is the cause of this issue. It was already a pain enough to get this to compile on ARM, as a had to do some special steps to get it to happen (described under additional context). I've just been banging my head against this for weeks now going through stack overflow and chatGPT. Please help, I would appreciate it so much.