ZeroMQ support for PubSub interaction with OpenFace

[NumesSanguis]

Thanks as always for creating OpenFace.

I see in your package folder that you've added ZeroMQ dependencies 5 months ago. My question is whether you've already decided you're going to use ZeroMQ to have OpenFace publish it's data to topics in a PubSub fashion?

I currently creating an opensource framework named FACSvatar - that extracts FACS data from a face using OpenFace, and then uses these data to animate an avatar. The only thing preventing this system from being real-time, is that my C++ coding is too bad to be able to stream your FACS data into the rest of my framework. Even if I would spend enough time on this, it would just be a temporary solution, because you'll probably finish a higher quality API for OpenFace.

I'm currently considering switching to ZeroMQ, so if OpenFace is also planning to use this, this would be a big motivation to indeed use ZeroMQ. Since OpenFace is a vital part of my framework.

--- FACSvatar motivation --- With FACSvatar I intend to improve Embodied Conversational Agents by giving agents more control over the facial animations of an avatar. My graduate program is ending in half a year, so I have to run experiments before then. Could you let me know the priority of the messaging API for OpenFace, because I have to use this to consider whether I can run real-time Human-Agent Interaction experiments or I have to focus on a more offline version. Thank you.

[TadasBaltrusaitis]

Hi,

Your project sounds really interesting!

I am indeed intending to use ZeroMQ to communicate between OpenFace and external applications (and there is already some experimental support for streaming head pose from a C# version of the code), but I would like to add support for C++.

However, while I will add the support eventually, it is currently not a high priority item and only will be added in 4-5 months I imagine (although this is not guaranteed), so depending on your timescales you might need a stop-gap solution until then.

Thanks, Tadas

[NumesSanguis]

Hello Tadas,

Thank you for your response. I successfully switched to ZeroMQ (as you can see here: https://github.com/NumesSanguis/FACSvatar/). My experience so far is that the library is very powerful.

Unfortunately, then waiting for ZeroMQ support in OpenFace is outside the time-scope of my master thesis. But I hope my project lives longer than that, so when FACS data through ZeroMQ is available, please let me know. Connecting our software should be very easy then.

Do you know how the project "SARA: the Socially Aware Robot Assistant" made use of OpenFace in real-time? Their paper shows their architecture: http://articulab.hcii.cs.cmu.edu/wordpress/wp-content/uploads/2016/08/SIGDIAL2016_Camera_Ready_CITATION.pdf

[TadasBaltrusaitis]

Hi,

Glad you managed to implement your communication through ZeroMQ, I will have a look at your code once I start moving OpenFace to ZeroMQ.

I did know that SARA used OpenFace :), I worked with them a bit to integrate it into their system during my time at CMU.

[NumesSanguis]

Most of my code is in Python though, but I hope it is still useful to you :)

How did they integrate it in real-time? By directly linking the C++ code? Is this code available somewhere?

[TadasBaltrusaitis]

I don't believe the code is available. I think they used a communication protocol similar to ZeroMQ, but I'm not 100% sure.

[NumesSanguis]

A professor in our lab, Hung-Hsuan Huang, just integrated ZeroMQ with OpenFace!

When you replace MainWindow.xaml.cs found here, with: https://github.com/NumesSanguis/FACSvatar/tree/json_message/openface Then, when you rebuild and start OpenFaceOffline.exe, it automatically starts ZeroMQ (NetMQ) in the background and publishes in the data format: [topic, timestamp, data_json].

Note that to compile this file, the developer needs to install the following two packages NetMQ and Newtonsoft.Json They can be download from the nuget portal site.

Also note that I use pubSocket.Connect(), because in my project I need many publishers - 1 subscriber (to make a M-proxy-N setup). So you might want to change this to pubSocket.Bind()

Even though it's not in C++, I hope you can integrate this in the project, because a lot of people would like to use OpenFace, and I guess this is better than waiting 5 months for a C++ version.

[TadasBaltrusaitis]

Thanks, I'll definitely have a look at it after releasing the new OpenFace version!

[NumesSanguis]

A video demonstrating FACSvatar with OpenFace 2.0 in real-time: https://www.youtube.com/watch?v=u8saBnznqe8

It still uses the C# GUI to send the data over ZeroMQ.