epfLLM / meditron

Meditron is a suite of open-source medical Large Language Models (LLMs).
https://huggingface.co/epfl-llm
Apache License 2.0
1.85k stars 169 forks source link

Accuracy calculation failure #30

Open sia-cerebras opened 8 months ago

sia-cerebras commented 8 months ago

I see that here in evaluate.py accuracy is calculated in two different ways. And there is an assert statements to make sure they match. In my tests, these assertion sometimes fails. Why is this assertion there? and what does it failure signify?

Thank you,