Evaluate with Huggingface Models

[ZackZikaiXiao]

Thank you for the great work. It's been really helpful.

When I tried to run the benchmark, I found no umls_2023ab_SMM4H-17_cid2name.json for en_norm and no umls_2023ab_MedDRA_cid2name.json for en_kg in the current version.

Besides, even though result.json is provided in evaluation_result/task_result, running evaluate_query.py still lead to invalid results, such as: "Yarn-Mistral-7b-128k": { "acquisition_1": { "total": 0, "ssm_correct": 0, "acc": -1 }, "acquisition_inc": { "total": 0, "ssm_correct": 0, "acc": -1 },

Are there any recent updates to the script I should know about?

Thanks again!

[JOHNNY-fans]

Thank you for your interest in our work.

For UMLS, since it requires a licence, we recommend that you download it from https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html and process it to get the dict of cid2name. I will upload this processing code later.

Regarding the lack of evaluation results, it's hard work to first check if the fields in the uploaded json file are the same as the ones we provided. We will also check the code in the last few weeks and then locate and fix it.

[JOHNNY-fans]

Thanks for your interest in our work. For the huggingface model, you just need to define your own “generate” function ( the "openai_api" function in

• MedOdyssey https://github.com/JOHNNY-fans/MedOdyssey/tree/main
• /src https://github.com/JOHNNY-fans/MedOdyssey/tree/main/src
• /inference https://github.com/JOHNNY-fans/MedOdyssey/tree/main/src/inference /utils.py , line 37 to line 50) to do the job of outputting a response based on a prompt. We recommend that you keep the arguments consistent.

ZikaiXiao @.***> 于2024年10月1日周二 00:07写道：

Thank you for the great work. It's been really helpful.

I've noticed test_query.py is set up for OpenAI. How can I use it for Huggingface models? Are there any recent updates to the script I should know about?

Thanks again!

— Reply to this email directly, view it on GitHub https://github.com/JOHNNY-fans/MedOdyssey/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIOLKN4WX2OMCYTQ6BBON3ZZFZMDAVCNFSM6AAAAABPDVJUYSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGU2TOMBZGY3DMMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>