search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
7.09k stars 1.91k forks source link

Configuring Azure OPENAI #2302

Open sudhanshu-myl opened 2 months ago

sudhanshu-myl commented 2 months ago

Hi, I want to evaluate gpt4o and other gpt related models on SWE bench dataset. Can someone guide by giving what steps to follow? I have azure openai access key but it is not getting configured. So, I am not able to access openai-based gpt models. Also, is there a way to club swe-bench git repo with lm-evaluation-harness repo, so that i can test llm model on swe-bench dataset. Does lm-eval supports azure openai key access support?

eyuansu62 commented 2 months ago

yep, it can support azure openai key

sudhanshu-myl commented 2 months ago

Can you please let me know the steps/code for it? I searched extensively on GitHub but could not find one. Thank you!

Get Outlook for iOShttps://aka.ms/o0ukef

From: eyuansu62 @.> Sent: Thursday, September 19, 2024 8:05:07 AM To: EleutherAI/lm-evaluation-harness @.> Cc: Sudhanshu Mishra @.>; Author @.> Subject: Re: [EleutherAI/lm-evaluation-harness] Configuring Azure OPENAI (Issue #2302)

yep, it can support azure openai key

— Reply to this email directly, view it on GitHubhttps://github.com/EleutherAI/lm-evaluation-harness/issues/2302#issuecomment-2359858953, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BLAGALVE2VZFJELNSN2JZZTZXIZ5XAVCNFSM6AAAAABOIW3OQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJZHA2TQOJVGM. You are receiving this because you authored the thread.Message ID: @.***>

eyuansu62 commented 2 months ago

you can request by setting the parameter: base_url="xxx".

gonz-mart commented 1 month ago

@sudhanshu-myl Did you manage to use it? I am also interested