OpenFace as a Controller

[olufolajimi]

I would like to use OpenFace as a control input for an external system, but this requires me to read the csv output into an arduino. I need to do this in real time but it seems the csv only gets generated when recording is stopped. Is there a way to make OpenFace output data into a pre-defined file (and perhaps make it overwrite the data so that I can just keep reading the same lines as my real-time data)?

Moreover, I don't need all the columns it generates so is there a way to make it generate less?

I'm working with the offline version on Windows 7.

[TadasBaltrusaitis]

It is possible to run OpenFace in realtime mode from a webcam (or even a video), however, this would require you to change the code by redirecting the recording to some sort of communication protocol (e.g. a messaging server or a TCP/IP connection). For a start you can have a look at the following lines in FeatureExtraction.cpp:

        // Setting up the recorder output
        open_face_rec.SetObservationHOG(detection_success, hog_descriptor, num_hog_rows, num_hog_cols, 31); // The number of channels in HOG is fixed at the moment, as using FHOG
        open_face_rec.SetObservationVisualization(visualizer.GetVisImage());
        open_face_rec.SetObservationActionUnits(face_analyser.GetCurrentAUsReg(), face_analyser.GetCurrentAUsClass());
        open_face_rec.SetObservationLandmarks(face_model.detected_landmarks, face_model.GetShape(sequence_reader.fx, sequence_reader.fy, sequence_reader.cx, sequence_reader.cy),
            face_model.params_global, face_model.params_local, face_model.detection_certainty, detection_success);
        open_face_rec.SetObservationPose(pose_estimate);
        open_face_rec.SetObservationGaze(gazeDirection0, gazeDirection1, gazeAngle, LandmarkDetector::CalculateAllEyeLandmarks(face_model), LandmarkDetector::Calculate3DEyeLandmarks(face_model, sequence_reader.fx, sequence_reader.fy, sequence_reader.cx, sequence_reader.cy));
        open_face_rec.SetObservationTimestamp(sequence_reader.time_stamp);
        open_face_rec.SetObservationFaceID(0);
        open_face_rec.SetObservationFrameNumber(sequence_reader.GetFrameNumber());
        open_face_rec.SetObservationFaceAlign(sim_warped_img);
        open_face_rec.WriteObservation();
        open_face_rec.WriteObservationTracked();

However, instead of recording the observation to a file you could replace them with a message to an external system.

Implementing such functionality has been on my to do list for a while, however, I never got round to it.

[olufolajimi]

I think this project using openface does that https://github.com/NumesSanguis/FACSvatar. They use ZeroMQ as the communication protocol. I've considered doing the same but it looks way too complicated for my intended application (and experience level).

I've also noticed that openface writes continuously to the csv it creates when it starts. I just need to repeatedly read some values from the last row (whichever one it is when reading). Do you think that would work? I'm trying to read it into a processing program then send related information over serial to the arduino.

[TadasBaltrusaitis]

I think the method you're proposing should work. OpenFace writes out the output every frame, the CSV file is generated on the fly rather than at the end of the processing. You will just need to be careful about not locking the file when reading it and when dealing with partially written lines. One option would be to copy the current state of the output csv file before processing, so as to avoid clashes.