search

ECNU-Cross-Innovation-Lab / LGCCT

(DOI: 10.3390/e24071010) LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition
GNU General Public License v3.0
7 stars 0 forks source link

Regarding cross corpus accuracy #1

Open VSK02101996 opened 2 years ago

VSK02101996 commented 2 years ago

How it will work for some kind of reviews or interviews (real world) ?

Trert111 commented 2 years ago

Our code is based on MUlT and supports CMU-MOSEI, CMU-MOSI and IEMOCAP. The preprocessed data can be obtained from the MUlT. If you want to use custom dataset, you should modify dataset.py and adjust the hyperparameters like hidden dim.

VSK02101996 commented 2 years ago

For custom data I need to preprocess using https://github.com/yaohungt/Multimodal-Transformer this right ?

Trert111 commented 2 years ago

For custom data I need to preprocess using https://github.com/yaohungt/Multimodal-Transformer this right ?

The benchmark datasets like MOSEI, MOSEI, and IEMOCAP can be downloaded directly in https://github.com/yaohungt/Multimodal-Transformer. For custom datasets, you should preprocess data following its standard practice or by yourself. This may require large modification and thus we recommend just referring to the code of our model and merging the model to your custom data pipeline. The default hyperparameter for the network can be referred in main.py.

VSK02101996 commented 2 years ago

What is meaning of standard practice like calculating mfcc , chroma etc or something else ? Also merging means creating model for custom data and merge that model with your model ; is it right ? @Trert111

VSK02101996 commented 2 years ago

Can you share the code how to do preprocessing of custom data ? @Trert111 . Atleast share how it is done on iemocap,mosi dataset ?

Trert111 commented 2 years ago

For MOSEI and MOSI, you can refer to CMU-MultimodalSDK. For IEMOCAP, you can refer to other related repositories. Our data is from here provided by MULT.

VSK02101996 commented 2 years ago

I don't understand this lines can you please elaborate ;

For custom datasets, you should preprocess data following its standard practice or by yourself. This may require large modification and thus we recommend just referring to the code of our model and merging the model to your custom data pipeline.

Trert111 commented 2 years ago

Since our code now only supports dataset of MOSEI, MOSEI, and IEMOCAP, the custom dataset may require rewriting a large portion of code, including dataset, dataloader, hidden dim, and even the training&testing procedures. Therefore, you can write your own code for your dataset and refer to what you need in our code. The core model is in modules.