JinFish
commented
4 months ago The dataset is randomly split into training, validation, and test sets. The reason for a significant gap between the test set and the validation set may lie in the fact that the dataset encompasses Twitter data from 2019 to 2022, leading to substantial distributional differences across the entire dataset, which in turn results in considerable disparities between the validation and test sets. In our paper, we also mentioned, "We collect tweets from each month in the years 2019, 2020, 2021, and 2022 to provide a more diverse and unbiased dataset, which also makes it more challenging."
In our paper, we use grid search, where the learning rate ranged from 1e-5 to 7e-5, and the batch size was between 8 and 32. Have you run experiments according to these hyperparameters and selected the best outcome?
In the README of this repository, there was an issue with missing images in the dataset linked through the Baidu Cloud (the cause was related to downloading the dataset through VSCode, which tends to interrupt when transferring a large number of files, resulting in some images being absent in the dataset). If you find that many images in the dataset you downloaded cannot be opened or have a size of 0K, please download the latest dataset according to the most recent README file.
cementbarrier
Dear Author, We are very interested in your work, so we retrained the model following your steps (by the way, it seems that your documentation does not specify the version of seqeval, and the version I installed is 1.2.2, I wonder if this has any impact on the replication). With the hyperparameters unchanged, after training for 15 rounds, the F1 score on the training set is 0.999, the F1 score on the validation set is 0.8088, and the F1 score on the test set in the final output is 0.7036. Below is part of the log output. Under the condition of random sampling, the results of the validation set and the test set should not differ so much. Is there any issue with the test set?