search

yaseen28 / MedDoc-Bot

A Chat Tool for Comparative Analysis of Large Language Models in the Context of Pediatric Hypertension Guidelines
Apache License 2.0
2 stars 0 forks source link

response time - dataset #1

Closed flckv closed 5 months ago

flckv commented 5 months ago

great work, thank you for sharing.

Can you clarify the datasets you used to measure the average response time in minutes, please? Pasted Graphic 4

I can see an Evaluation dataset you shared of size 132KB https://github.com/yaseen28/MedDoc-Bot/blob/main/Dataset/Benchmark%20Dataset%20For%20Evaluation.pdf - was this one used to create Figure 3? I can see that probably one of the document in the test was: https://academic.oup.com/eurheartj/article/43/35/3290/6633855 that is equivalent to document of size 981 KB MedDoc-Bot/Dataset/Original Pediatric_HTN_Guideline.pdf

yaseen28 commented 5 months ago

Hi thank you for writing :)

yes i used the https://github.com/yaseen28/MedDoc-Bot/blob/main/Dataset/Benchmark%20Dataset%20For%20Evaluation.pdf to measure the average response time. I posted four questions from each group and calculated the time taken by each model to generate a response for every single question. Then, I averaged these times and plotted the graph above. Additionally, the PDF document from the European Heart Journal (https://academic.oup.com/eurheartj/article/43/35/3290/6633855)) was uploaded to our dashboard and queried with the benchmark dataset. For more details, please refer to the article: arXiv:2405.03359.