runnanzhou
commented
4 years ago Hi Adarsh,
Thank you! I will do it.
Best wishes, Runnan Zhou
On Apr 19, 2020, at 11:53 AM, Adarsh Pyarelal notifications@github.com wrote:
External Email
@runnanzhou https://github.com/runnanzhou - you might have done parts of this process already, but I wanted to document the task in detail for you, it might help. Apologies, I had meant to do this earlier, but am just getting around to it now.
We need to develop an executable that can process input webcam live video (or video files or images from disk) and output facial action units that are automatically detected using OpenFace in a JSON format.
Here are the implementation steps. To start with, we will implement the 'static' method, that requires just the frames and not a video on disk or in memory (the dynamic method is more accurate since it performs AU normalization, but is more complicated to implement).
Update src/WebcamSensor.cpp and src/WebcamSensor.h to prefix the OpenFace included headers with OpenFace/ i.e. change the following line:
include "GazeEstimation.h"
to
include <OpenFace/GazeEstimation.h>
Do the same substitution for LandmarkCoreIncludes.h, SequenceCapture.h, VisualizationUtils.h, and Visualizer.h in src/WebcamSensor*.cpp.
Create a file called src/AUSensor.cpp, create an int main(...) function in it.
Add a line at the top of src/AUSensor.cpp to include the WebcamSensor.h header file. Inside the main function, add a line that creates an object that is an instance of the WebcamSensor class.
Modify the arguments attribute in WebcamSensor.h, adding the string -au_static to the vector.
Create a while loop in the main function that keeps calling the get_observation method of the WebcamSensor class, until the SIGTERM signal is received, at which point the loop should exit.
Add the line add_subdirectory(external/OpenFace) at some point before the call to add_subdirectory(src) in tomcat/CMakeLists.txt
Add a line called add_executable(ausensor AUSensor.cpp) in src/CMakeLists.txt.
Immediately after that line, add the line
target_link_libraries(ausensor PUBLIC OpenFace) You might need to add the line
target_include_directories(ausensor ${openface_include_dirs}) as well.
Navigate to the build/ directory in the tomcat root directory and execute
cmake .. make -j ausensor Test the functionality of the executable by running ./bin/ausensor (from within the build directory). You should see face landmarks being tracked, as well as the gaze.
At this point, you will have recreated the original functionality of the WebcamSensor class when it was integrated into the runExperiment executable. Once you have verified this, then modify src/WebcamSensor.cpp to get the action units by looking at https://github.com/TadasBaltrusaitis/OpenFace/blob/ad1b3cc45ca05c762b87356c18ad030fcf0f746e/exe/FeatureExtraction/FeatureExtraction.cpp#L190-L258 https://github.com/TadasBaltrusaitis/OpenFace/blob/ad1b3cc45ca05c762b87356c18ad030fcf0f746e/exe/FeatureExtraction/FeatureExtraction.cpp#L190-L258 and adding the appropriate portions of that code to src/WebcamSensor.cpp to perform AU extraction.
Once you have a minimal setup going, then modify the program to output the AUs to the standard output in JSON format. You can use the nlohmann-json library for this (see src/Mission.cpp for an example of usage). The relevant attributes of the RecorderOpenFace class to work with are au_intensities and au_occurrences: https://github.com/ml4ai/tomcat/blob/08115c44d42ad552a2fac08d1f566a785a778c66/external/OpenFace/lib/Utilities/include/RecorderOpenFace.h#L189-L190 https://github.com/ml4ai/tomcat/blob/08115c44d42ad552a2fac08d1f566a785a778c66/external/OpenFace/lib/Utilities/include/RecorderOpenFace.h#L189-L190. Start with a simple JSON format, then we can see how to massage it into the form that the TA3 testbed expects. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ml4ai/tomcat/issues/113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJFHXEUBPUST64ZPESO7IVTRNNCBJANCNFSM4ML452DA.
adarshp
shreeyajain
@runnanzhou - you might have done parts of this process already, but I wanted to document the task in detail for you, it might help. Apologies, I had meant to do this earlier, but am just getting around to it now.
We need to develop an executable that can process input webcam live video (or video files or images from disk) and output facial action units that are automatically detected using OpenFace in a JSON format.
Here are the implementation steps. To start with, we will implement the 'static' method, that requires just the frames and not a video on disk or in memory (the dynamic method is more accurate since it performs AU normalization, but is more complicated to implement).
[x] Update
src/WebcamSensor.cppandsrc/WebcamSensor.hto prefix the OpenFace included headers withOpenFace/i.e. change the following line:to
[x] Do the same substitution for
LandmarkCoreIncludes.h,SequenceCapture.h,VisualizationUtils.h, andVisualizer.hinsrc/WebcamSensor*.cpp.[x] Create a file called
src/AUSensor.cpp, create anint main(...)function in it.[x] Add a line at the top of
src/AUSensor.cppto include theWebcamSensor.hheader file. Inside the main function, add a line that creates an object that is an instance of the WebcamSensor class.[x] Modify the
argumentsattribute inWebcamSensor.h, adding the string-au_staticto the vector.[x] Create a while loop in the main function that keeps calling the
get_observationmethod of the WebcamSensor class, until the SIGTERM signal is received, at which point the loop should exit.[x] Add the line
add_subdirectory(external/OpenFace)at some point before the call toadd_subdirectory(src)intomcat/CMakeLists.txt[x] Add a line called
add_executable(ausensor AUSensor.cpp)insrc/CMakeLists.txt.[x] Immediately after that line, add the line
[x] You might need to add the line
as well.
[x] Navigate to the
build/directory in the tomcat root directory and execute[x] Test the functionality of the executable by running
./bin/ausensor(from within thebuilddirectory). You should see face landmarks being tracked, as well as the gaze.At this point, you will have recreated the original functionality of the WebcamSensor class when it was integrated into the
runExperimentexecutable. Once you have verified this, then modifysrc/WebcamSensor.cppto get the action units by looking at https://github.com/TadasBaltrusaitis/OpenFace/blob/ad1b3cc45ca05c762b87356c18ad030fcf0f746e/exe/FeatureExtraction/FeatureExtraction.cpp#L190-L258, and adding the appropriate portions of that code tosrc/WebcamSensor.cppto perform AU extraction.nlohmann-jsonlibrary for this (seesrc/Mission.cppfor an example of usage). The relevant attributes of theRecorderOpenFaceclass to work with areau_intensitiesandau_occurrences: https://github.com/ml4ai/tomcat/blob/08115c44d42ad552a2fac08d1f566a785a778c66/external/OpenFace/lib/Utilities/include/RecorderOpenFace.h#L189-L190. Start with a simple JSON format, then we can see how to massage it into the form that the TA3 testbed expects.