search

kbressem / medAlpaca

LLM finetuned for medical question answering
GNU General Public License v3.0
491 stars 58 forks source link

Compare to BioMedLM #21

Open rdenaux opened 1 year ago

rdenaux commented 1 year ago

Congrats on the interesting project and good results! I was wondering if you have considered adding a comparison or finetuning BioMedLM (available from HuggingFace) on your dataset. From their blogpost, they claim a score of 50.3% in MedQA-USMLE. It's not clear how this relates to your benchmark scores, also for USMLE.

kbressem commented 1 year ago

Thanks for the suggestion. I'll probably run this comparison, once I am satisfied with the re-implementation of the training.