search

princeton-nlp / SWE-bench

[ICLR 2024] SWE-Bench: Can Language Models Resolve Real-world Github Issues?
https://www.swebench.com
MIT License
1.81k stars 311 forks source link

Fix pip/python paths in context_manager.py (fixes #55) #84

Closed thakkarparth007 closed 5 months ago

thakkarparth007 commented 6 months ago

For many repos people have been complaining about wrong conda version being selected. Turns out that's happening because, despite activating the right environment, somehow the PATH env var doesn't get updated correctly, due to which "pip" refers to ~/miniconda/bin/pip instead of /bin/pip. This fixes that issue. The evals should work after that.

brombaut commented 5 months ago

I've tried checking if this works, but I'm still running into the issue described in #77

john-b-yang commented 5 months ago

Hi @thakkarparth007 thanks so much for the contribution, and also thanks @brombaut for the additional comments.

Carlos and I have just released a report detailing our recent efforts to correct the issues that people are have with evaluation.

The latest version of SWE-bench captures several of the changes proposed here, along with additional efforts to fix the benchmark.

If you have been having issues w/ SWE-bench, I'd recommend re-running evaluation with the latest version of SWE-bench to see if the latest fixes are working. If not, feel free to follow up with an issue.

Thanks again for contributing, really appreciate these efforts!