search

yixuantt / FinEntity

Code and resource of "FinEntity: Entity-level Sentiment Classification for Financial Texts" EMNLP 2023
10 stars 1 forks source link

Reproduce the results of Table 3 for FinBERT-CRF #1

Closed grphoto closed 6 months ago

grphoto commented 6 months ago

Thanks for your research and it is a breakthrough to have a entity level sentiment dataset for financial domain.

After reading the paper, I try to reproduce the results of Table 3 for FinBERT-CRF. But I got train loss of nan (probably infinite) when running bert_crf_train.ipynb without making any changes. Would you mind advising the way to reproduce it?

Another question related to this is whether the result is on unseen testing set or validation set, which is used for hyperparameter search, because there is only "validate_dataset" inside bert_crf_train.ipynb without testing set.

My Library version torch: 2.2.2+cu121 transformers: 4.39.3

Thank you

yixuantt commented 6 months ago

Hi, thanks for your kind words. For this project, here is my setting for your reference (I just test and it runs)

transformers             4.40.2
torch                2.3.0+cu121

For the reproduction of Table 3, I pre-divided the training dataset and the test dataset at a ratio of 8:2 and stored them separately. However, for the simplicity in the code repository, I only uploaded the complete FinEntity dataset and modified some of the code. Thus, Table 3 is tested on an unseen testing set (a subset of FinEntity).

I hope this information will be helpful to you.

grphoto commented 6 months ago

Thank you for your prompt reply I think it is the problem of the machine. I am able to run bert_crf_train.ipynb inside Google Colab.