search

cosanlab / py-feat

Facial Expression Analysis Toolbox
https://py-feat.org/
Other
261 stars 71 forks source link

Refactor FEX class. #58

Closed jcheong0428 closed 3 years ago

jcheong0428 commented 3 years ago

Remove sub-classes Facet, OpenFace, Affdex and consolidate into a single Fex class.

4 key features.

  1. Detector class: Detects facial landmarks, AUs, emotions, from a face image or video.
  2. Fex class: Main data class for manipulation
  3. Extractor class: Extracts features from a Fex object.
  4. Plot & visualization functions.

Sample usage.

1. Detector class.

detector = Detector()
# save detection results to a file or a Fex output.
fex = detector.detect_image(filename=imageFile)
fex =detector.detect_video(filename=videoFile)

2. Fex class

2-1. Initializing.

# load directly from image 
fex = detector.detect_image(imageFile)
# load from Facet, OpenFace, or Affectiva files. 
fex = Fex()
fex = fex.read_facet(filename=facetfile)
fex = fex.read_openface(filename=facetfile)
fex = fex.read_affectiva(filename=facetfile)

or initialize with info and read file. 

fex = Fex(filename=filename)
fex = fex.read_file()

2-2. Attributes.

functionals fex.aus: Dataframe of action units. fex.emotions: Dataframe of emotions

structurals fex.landmarks: Dataframe of facial landmarks fex.facebox: Dataframe of face bounding box.

designs fex.time: Dataframe of time info. fex.design: Dataframe of design matrix info.

all fex.data: Dataframe of all columns.

meta fex.meta: meta info such as filenames, fps, etc.

aus, emotions, landmarks, facebox, time, sessions should all just be a function that uses column names for each to slice fex.data. Column names should be stored under fex.au_columns, fex.emotion_columns, fex.landmark_columns, fex.facebox_columns, fex.time_columns, fex.design_columns.

2-3 Methods.

All methods are implemented under the Fex class but they can also be aggregated using the Extractor class. extract_mean extract_min extract_max extract_summary: Wrapper to extract mean, min, and max. extract_wavelet extract_multi_wavelet extract_boft calc_pspi

3. Extractor class (TBD on whether to scrap)

The extractor class simply aggregates info from different Fex class. For example collect all features across files:

extractor = Extractor()
for file in imagefiles:
    fex = Fex.read_facet(file)
    extractor.min(fex)

4. Visualization & plotting.

Static plotting Interactive plotting.

Reference: https://github.com/fexmetrica/fex-metrica/blob/master/fexSDK/src/fexc.m

jcheong0428 commented 3 years ago

Trying to create a doc so everyone can be on the same page. Feel free to edit or comment.