search

princeton-nlp / SWE-bench

[ICLR 2024] SWE-bench: Can Language Models Resolve Real-world Github Issues?
https://www.swebench.com
MIT License
2.01k stars 350 forks source link

Want apptainer support #242

Closed HotineXie closed 1 week ago

HotineXie commented 3 weeks ago

Describe the feature

Run SWE-bench evaluation requires a lot of computing resources, so it may be a good idea to run it on supercomputers. However, docker is usually not supported on supercomputers. Is it possible for us to add support for the apptainer?

Potential Solutions

Some python pkg support the interactive with apptainer/singulariy like singularity python. If we use their API to replace the docker pkg, we may be able to use apptainer to maintain the exec environment.

john-b-yang commented 3 weeks ago

Hi @HotineXie, thanks for this suggestion. This could be a great way to augment SWE-bench.

Unfortunately, we don't plan on actively supporting extensions to different containerization software. However, I have marked this issue with the future and help wanted labels.

I'll leave this issue open for a week in case anyone who comes across this issue would like to help push this direction.

john-b-yang commented 1 week ago

Closing this due to inactivity, but if anyone comes across this and decides to take up this issue and make a PR for it, please feel free to re-open.