InstructionERC

🎥 Overview

This repository contains the open-sourced official implementation of our work InstructERC:

InstructERC: Reforming Emotion Recognition in Conversation with a Retrieval Multi-task LLMs Framework

If you find this repo helpful, please cite the following paper:

@article{lei2023instructerc,
  title={Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework},
  author={Lei, Shanglin and Dong, Guanting and Wang, Xiaoping and Wang, Keheng and Wang, Sirui},
  journal={arXiv preprint arXiv:2309.11911},
  year={2023}
}

Introduction

In this study, we propose a novel approach, namely InstructERC, to reformulates the ERC task from a discriminative framework to a generative framework based on LLMs. InstructERC has two significant contributions: Firstly, InstructERC introduces a simple yet effective retrieval template module, which helps the model explicitly integrate multi-granularity dialogue supervision information by concatenating the historical dialog content, label statement, and emotional domain demonstrations with high semantic similarity. Furthermore, we introduce two additional emotion alignment tasks, namely speaker identification and emotion prediction tasks, to implicitly model the dialogue role relationships and future emotional tendencies in conversations. Our LLM-based plug-and-play plugin framework significantly outperforms all previous models and achieves comprehensive SOTA on three commonly used ERC datasets. Extensive analysis of parameter-efficient and data-scaling experiments provide empirical guidance for applying InstructERC in practical scenarios. Our code will be released after blind review.

🍯 Overall Framework

🎯 Quick Start

This repo consists of following files:

.
├── checkpoint
├── code
│   ├── data_process_mixed.py
│   ├── data_process_plain.py
│   ├── data_process.py
│   ├── data_utils
│   ├── main_new.py
│   ├── train_and_inference_Mixed.sh
│   ├── train_and_inference_Plain.sh
│   └── train_and_inference_Uni.sh
├── data
│   ├── EmoryNLP
│   ├── iemocap
│   └── meld
├── demo
│   └── demo.ipynb
├── envs
│   └── requirements.txt
├── experiments
├── file_structure.txt
├── LLM_bases
│   ├── Bloom-560m
│   ├── ChatGLM
│   ├── ChatGLM2
│   ├── LLaMA
│   └── LLaMA2
├── original_data
│   ├── dailydialog
│   ├── EmoryNLP
│   ├── iemocap
│   ├── meld
│   └── peek_of_dataset.ipynb
└── README.md

As shown in the tree-like structure above, InstructERC consists of the following folders: code, data, demo, envs, LLM_bases, and original_data.

• The checkpoint folder is created to storage InstructERC'checkpoint. (Pertraining Checkpoint, Supervised Finetuning Checkpoint at each epoch)

• The code folder contains all the executable code for InstructERC, including data processing scripts such as data_process.py (mixed, plain), the main program file main_new.py, data_utils, and the bash script train_and_inference_Uni.sh (Plain, Mixed) that controls the entire workflow.

• The data folder stores the data that has been processed by the data_process.py script from the original data. This processed data can be directly fed into the LLM (Large Language Model) as input.

• The demo folder contains the script demo.ipynb, which can be loaded and run directly on the LLM that has been finetuned on specific data. It can be opened and run using Jupyter Notebook.

• The envs folder contains the relevant dependencies, environments, and libraries required for this project. We strongly recommend using Docker and creating a new Conda virtual environment to avoid affecting your existing environment and files.

• The experiments folder is created to storage the result under different experiment's settings.

• The LLM_bases folder stores the original models provided by the official LLMs. These models can be downloaded from Hugging Face.

• The original_data folder contains the widely used datasets processed by the COSMIC team, including IEMOCAP, MELD, EmoryNLP, and dailydialog. We write a script for you to have a peek of these datasets, namely peek_of_dataset.ipynb.

Dependencies

We suggest you create a docker environment for InstructERC to ensure that your previous systems, libraries and files are not effected. Make sure your Devtoolset-8-toolchain' version align with us:

yum install devtoolset-8-toolchain
yum install gcc 9.3.1

General Setup Environment:

• Python 3
• NumPy
• PyTorch (currently tested on version 2.0.0)
• Transformers (version 4.30.2, unlikely to work with a different version)

InstructERC Setup Environment:

cd ./InstructERC/envs/
pip3 install -r requirements.txt

LLMs download

Follow this link.

ONLY Validate Our Work

Due to Meituan's code review process, the public release date of the model parameters is unpredictable. However, based on our tests on other machines, we have achieved performance results that fluctuate by ±0.5 compared to the data presented in the paper. Additionally, for the implementation of the demonstration, you can refer to the following link: https://github.com/UKPLab/sentence-transformers

Completely repeat all our work

cd ./InstructERC

To reproduce the results, we have three pipelines available.

• If you want to reproduce the Main Result Reproduction, you can run the train_and_inference_Uni.sh

bash train_and_inference_Uni.sh

The Shellparameter that controls the mainprocess: Flag

the value of which is 0 or 1. 
The mainprocess will interrupt when flag is 0

The hyperparameters you need setting:

1.MODEL_NAME (selections: ChatGLM, ChatGLM2, LLaMA, LLaMA2)
# MODEL_NAME determines on which model base InstructERC will be fine-tuned.

2.Experiments_setting (selections: LoRA, All-parameters)
# The Experiments_setting parameter determines whether it is full parameter fine-tuning or efficient parameter fine-tuning.

3.dataset (selections: IEMOCAP, MELD, EmoryNLP)
# The specific dataset you want InstructERC to finetune on.

4.accumulations (type:int)
# Due to the limitations of the GPU, we have chosen the method of gradient accumulation for fine-tuning.

5.graphics_card (type:int)
# The graphics_card represents the number of graphics cards you use when fine-tuning.

Notes: batch size = graphics_card * accumulations

The remaining subprocesses determined by these hyperparameters are designed to conduct different experiments.

• If you want to reproduce the Unified dataset Experiment, you can run the train_and_inference_Mixed.sh

bash train_and_inference_Mixed.sh

Compared to train_and_inference_Uni.sh, you should overlook the hyperparameter dataset due to the unified dataset including all ERC datasets.

• If you want to reproduce the LoRA+Backbone Experiment, you can run the train_and_inference_Plain.sh

bash train_and_inference_Mixed.sh

📋 Result:

Main Result

All Parameters vs Parameter Efficiency

In order to investigate the effect of different parameter fine-tuning methods on the ERC task, we conducted comparative experiments in Table 2.

A.1 Unified dataset labeling

We continue to use the previous datasets IEMOCAP, MELD, and EmoryNLP. In accordance with The Feeling Wheel [^1] proposed in 1982, as shown in Figure 2, we align all emotional labels of three datasets under this standard, the details of which are shown in Table 3. After completing the label mapping, there are a total of 9 kinds of emotional labels, which are joyful, sad, neutral, mad, excited, powerful, fear, peaceful, and disgust.

Figure2: The Feeling Wheel[^1]

A.2 Unified dataset Experiment

We still utilize the LoRA method in PEFT to train InstructERC on the unified dataset, and the training results are evaluated on the three datasets respectively. Meanwhile, we design total mix and ratio mix experiments to explore the impact of different data mixing strategies and data quantities on the model. On below basis, we further explore the impact of data sampling ratio on the model's performance. The details are shown in the Table 5, a more intutive presentation is shown in Figure 6.

[^1]: Willcox, K. (1982). The Feeling Wheel. Journal of Counseling & Development, 61(3), 191-193.