search

justinphan3110 / SciFive

SciFive: a text-text transformer model for biomedical literature
https://arxiv.org/abs/2106.03598
MIT License
89 stars 16 forks source link

pubmed-pmc-large version seems wrong #10

Closed qiuhaolu closed 2 years ago

qiuhaolu commented 2 years ago

Thanks for sharing the models. I found that the pubmed-pmc-large version (https://huggingface.co/razent/SciFive-large-Pubmed_PMC/tree/main) seems wrong as the fine-tuning results on MEDNLI are drastically worse than T5-large (acc 0.67+ vs 0.82+). However, the pubmed-large version gives good results.

Could you confirm this version?

justinphan3110 commented 2 years ago

Hi @qiuhaolu , can you share the notebook to reproduce these fail results?

qiuhaolu commented 2 years ago

Thanks @justinphan3110. The model should be right.