The AU intensities generated by OpenFace is inconsistent with the AU intensities manually marked

[Au-Chak-Hon]

Hi,

I used OpenFace for AU intensity estimation and I found that the AU intensities generated by OpenFace is very different from the manually marked AU intensities.

The dataset used is DISFA and the subject used is SN001.

The command used to run the AU intensity estimation is: "./../../OpenFace_2.2.0_win_x64\FeatureExtraction.exe" -aus -f "./../../DISFA/SN001/video\LeftVideoSN001_comp.avi" -out_dir "./../../DISFA/SN001/csv_vid" and "./../../OpenFace_2.2.0_win_x64\FeatureExtraction.exe" -aus -f "./../../DISFA/SN001/video\RightVideoSN001_comp.avi" -out_dir "./../../DISFA/SN001/csv_vid"

Then, the distribution looks like the following: Using LeftVideo: Using RightVideo: Python Code for showing the distribution:

import pandas as pd
import os
import numpy as np

data_root_path = './../../DISFA/SN001'
csv_path = data_root_path + '/csv_vid'
openface_path = './../../OpenFace_2.2.0_win_x64'

labels = np.array([1,2,4,5,6,9,12,15,17,20,25,26])
AUs1 = np.zeros([12,6])
AUs2 = np.zeros([12,6])
AUs3 = np.zeros([12,6])
AUs_real = np.zeros([12,6])

if True:  
  csv_list = pd.read_csv(os.path.join(csv_path,('RightVideoSN001_comp.csv')),sep=', ')
  for j in range(len(csv_list)):
    frame = csv_list.loc[j,['AU01_r','AU02_r','AU04_r','AU05_r','AU06_r','AU09_r','AU12_r','AU15_r','AU17_r','AU20_r','AU25_r','AU26_r']]
    for i in range(len(frame)):
      _value1 = math.floor(frame[i])
      _value2 = math.ceil(frame[i])
      _value3 = round(frame[i])
      AUs1[i][_value1] += 1
      AUs2[i][_value2] += 1
      AUs3[i][_value3] += 1

  idx = 0
  for list in sorted(os.listdir(os.path.join(data_root_path,'AU'))):
    _l = pd.read_csv(os.path.join(data_root_path,'AU',list),sep=',',header=None)
    for j in range(len(_l)):
      _f = _l.loc[j,1]
      _value = int(_f)
      AUs_real[idx][_value] += 1
    idx += 1

  print('Estimated Distribution (floor):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs1[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs1[x][0]) + '|' + "{:^6.0f}".format(AUs1[x][1]) + '|' + "{:^6.0f}".format(AUs1[x][2]) + '|'
          + "{:^6.0f}".format(AUs1[x][3]) + '|' + "{:^6.0f}".format(AUs1[x][4]) + '|' + "{:^6.0f}".format(AUs1[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))

  print('Estimated Distribution (ceil):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs2[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs2[x][0]) + '|' + "{:^6.0f}".format(AUs2[x][1]) + '|' + "{:^6.0f}".format(AUs2[x][2]) + '|'
          + "{:^6.0f}".format(AUs2[x][3]) + '|' + "{:^6.0f}".format(AUs2[x][4]) + '|' + "{:^6.0f}".format(AUs2[x][5]) + '|' + "{:^6.0f}".format(_AUTotal)) 

  print('Estimated Distribution (round):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs3[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs3[x][0]) + '|' + "{:^6.0f}".format(AUs3[x][1]) + '|' + "{:^6.0f}".format(AUs3[x][2]) + '|'
          + "{:^6.0f}".format(AUs3[x][3]) + '|' + "{:^6.0f}".format(AUs3[x][4]) + '|' + "{:^6.0f}".format(AUs3[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))

  print()
  print('Real Distribution:')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs_real[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs_real[x][0]) + '|' + "{:^6.0f}".format(AUs_real[x][1]) + '|' + "{:^6.0f}".format(AUs_real[x][2]) + '|'
          + "{:^6.0f}".format(AUs_real[x][3]) + '|' + "{:^6.0f}".format(AUs_real[x][4]) + '|' + "{:^6.0f}".format(AUs_real[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))

Also, it is found that the positions of the AU intensities does not match. Take AU1 as an example, the detected frame position for intensity 1 is approximately at frame 3000 to 3200, but the manually marked frame position for intensity 1 is approximately at frame 1600

Frames detected with intensity 1 in the left vieeo: Frames detected with intensity 1 in the right vieeo: The actual frames with intensity 1:

I wonder if I did something wrong, or did I misunderstood something. Please help. Thank you.