search

MAGICS-LAB / DNABERT_2

[ICLR 2024] DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome
Apache License 2.0
254 stars 59 forks source link

Train Dev Test Splits for GUE Benchmark Differ from Reported Paper Splits #52

Closed o-turbitt-mdc closed 10 months ago

o-turbitt-mdc commented 11 months ago

Hi Zhihan,

Thank you very much for your hard work on this repo, much appreciated!

I have been trying to recreate the results of the DNABERT2 on the COVID variant prediction task from the GUE benchmark and I have noticed a discrepancy between the reported values in the paper for the train, validation, and test splits.

In the paper, it is reported that the breakdown for these splits is: 77669 / 7000 / 7000. However, using the files provided here, I get a breakdown for the splits as: 73335 / 9168 / 9168.

I haven't checked any of the other benchmark datasets but this may be an issue with others too.

Is it possible for you to fix the datasets provided to the correct splits given in the paper? This will allow others to recreate and validate your results.

Zhihan1996 commented 10 months ago

Thanks for the comments! It might be a typo in the paper. I will double-check the statistics and update the paper soon. Thanks for the catch again!

ZhengPeng7 commented 8 months ago

Is there any update on this? It's important to the comparison.