All-Hands-AI / OpenHands

🙌 OpenHands: Code Less, Make More

https://all-hands.dev

MIT License

31.4k stars 3.62k forks source link

add SWE-bench dev set support #3477 #3478

Open jatinganhotra opened 1 month ago

jatinganhotra commented 1 month ago

What problem or use case are you trying to solve? Linked to feat: add SWE-bench fullset support #3477

3477 adds instance level images for `test` sets. It would be awesome if you could add 'princeton-nlp/SWE-bench', split='dev' and 'princeton-nlp/SWE-bench_Lite', split='dev') as well.

Describe the UX of the solution you'd like

Do you have thoughts on the technical implementation?

Describe alternatives you've considered

Additional context

jatinganhotra commented 2 weeks ago

Copying the link to discussion from #3477 here: https://github.com/All-Hands-AI/OpenHands/pull/3477/files/a8e35be0b7f95b9018c6d0976615397500f287a1#r1741178721 as it was closed and merged.

@xingyaoww we're still waiting for SWE-Bench authors to resolve an issue, which is blocking adding 'dev' set evaluation. I reached out to them and will share here when they have resolved it.

All-Hands-AI / OpenHands

add SWE-bench dev set support #3477 #3478

3477 adds instance level images for test sets. It would be awesome if you could add 'princeton-nlp/SWE-bench', split='dev' and 'princeton-nlp/SWE-bench_Lite', split='dev') as well.

3477 adds instance level images for `test` sets. It would be awesome if you could add 'princeton-nlp/SWE-bench', split='dev' and 'princeton-nlp/SWE-bench_Lite', split='dev') as well.