search

zinengtang / TVLT

PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral)
MIT License
120 stars 13 forks source link

Finetuning for emotion analysis but nan output #9

Open Changezi001 opened 1 year ago

Changezi001 commented 1 year ago

Hi,

I have downloaded the CMU-MOSEI dataset, and I am fine-tuning TVLT for emotion analysis on the MOSEI dataset. Although I could not obtain the labels_emotion folder in the downloaded zip file, I have compiled my own emotion label file using the [timestamp file] (http://immortal.multicomp.cs.cmu.edu/timestamped_links/) from CMU, but when I use the [fine-tuning script](url) to fine-tune it on CMU-MOSEI, the model outputs nan values after some iterations. Could you shed some light on what could be wrong?

zinengtang commented 1 year ago

Could you explain how did you convert the values to the labels?

Changezi001 commented 1 year ago

I just took the mosei.csv file and removed columns 1:5. But, before removing col 1:5, I changed column 0, i.e., id column, to FileName column by: column0 + '' + column_2

Changezi001 commented 1 year ago

And in moseiemo_dataset.py, line number 78, I changed it to emolist = np.array(self.labels[self.labels['FileName']==self.keys[index]].iloc[0, 1:] > 0.0), and that gives me boolean labels