search

m42-health / med42

MIT License
36 stars 0 forks source link

Performance on BioASQ factoid questions #1

Open Neelectric opened 9 months ago

Neelectric commented 9 months ago

First of all, congratulations on your model! The benchmark scores look impressive. I downloaded it from HF and will try to quantize and play around with it soon.

In the vein of the model's high performance on QA tasks like MedQA and MedMCQA, I wondered if there were any plans to benchmark the model on BioASQ? Particularly I would be curious about the model's performance on factoid questions, for example from BioASQ5b: http://participants-area.bioasq.org/datasets/

This is a common QA dataset, so given M42's encouraging results on other QA tasks, I was curious how well it does on its factoid questions. Thank You!

m42pimentel commented 9 months ago

@Neelectric , many thanks for your interest in our model and for your comments. That is an excellent suggestion. We will attempt to access the dataset and benchmark the model on it.