Real-time usage with named pipe

[mnalli]

Hello!

First of all, thank you for this great project.

I am currently trying to implement a facial expression sonification application, which basically means that I have to translate AUs into sound in a meaningful way. For non real-time usage this is not a big problem, as I can simply read the .csv file to get the information I need to generate the audio track. My troubles are related to real-time usage (translate AUs extracted live from a webcam into sounds).

Reading some related issues, it seems that the solutions outlined (directly or indirectly) seem to be of two kinds:

• Using a messaging library to stream (maybe over the network) the real-time feature to another process, for it to consume the features: #375, #409, #492
  • As far as I understand, there is a modified version of OpenFace that is able to do this leveraging the ZeroMQ library. Apparently, this works only on Windows, but it would still be possible to use its docker image to run it on Linux and send real-time FACS data to our (e.g. python) program running outside of the container.
• Using a named pipe, so that OpenFace could write its csv features to the pipe and our client process could just read them from there. This sort of solution seems to be indirectly mentioned in these issues: #409, #546.

Right now I'm trying out the second solution, as it seems it's the one that would work more quickly. I did a dirty hack to do this: I created a named pipe called pipe.csv and called FeatureExtraction with the argument -of pipe, so that FeatureExtraction would write the csv results to the pipe. This seems to work for now, and I'm able to get the features real-time in another python process running outside of the (Ubuntu) container.

I guess the best way to do this would be to modify the code to add a --pipe argument that would instruct FeatureExtraction to specify a named pipe where to output the real-time features. I think it should be also possible to do something like this on Windows.

For now I think I will stick with this dirty hack, also because, as I'm running Debian, I can't really compile OpenFace. I'm using the Ubuntu docker, where the pipe file is in a directory shared between the container and the system. I think I will install Ubuntu soon anyways, so maybe I will try to modify the code.

Does anyone have suggestions? Is maybe someone aware of some other repositories or solutions that would help me to handle this appropriately?

Best, Michele

[ThomasJanssoone]

Hello,

I just pushed on my forked branch the code I did to integrate ZeroMQ into openFace. It is simple an adaptation of the featureExtractor that stream the AUs values, gaze and head rotation to a python script

Please feel free to test this and use it

@TadasBaltrusaitis I would like to know if it is possible to merge this directly into your repo

Best

[mnalli]

Hello,

Thank you very much, it seems really interesting. Is it supposed to work for Windows, macOS and Linux?

Best

[fabiodrbarros]

hey @ThomasJanssoone , where i can find root_dir,"..\..\build\bin\AuExtractionMQ.exe in your branch?