Open JiaangL opened 8 months ago
I guess the hyper-parameter settings are changed during experiments on Further Analyses in our paper, I will release some running logs recording the correct training and inference processes.
Logs could be downloaded using Google Drive. Thanks for your attention, please feel free to contact us whenever you have other questions.
Hi, thanks for your nice work. I've downloaded the repo and tried to reimplement the results from the paper. But I've got different results on the Hallucination dataset with GPT2-XL. I'm running with the default hyperparameter setting in
\config
. The outputs are as follows:The Locality (TRR) and ARR are different from the original paper (12.31 vs 17.45, 8.63 vs 2.66). Is there anything I did wrong? Moreover, after checking the data, I find only 516 accurate outputs in Hallucination. Could you double-check the dataset statistics, by checking the shape of the accurate dataset in this line? Thanks in advance!