john-b-yang
commented
4 days ago Hi @HotineXie, thanks for this suggestion. This could be a great way to augment SWE-bench.
Unfortunately, we don't plan on actively supporting extensions to different containerization software. However, I have marked this issue with the future and help wanted labels.
I'll leave this issue open for a week in case anyone who comes across this issue would like to help push this direction.
Describe the feature
Run SWE-bench evaluation requires a lot of computing resources, so it may be a good idea to run it on supercomputers. However, docker is usually not supported on supercomputers. Is it possible for us to add support for the apptainer?
Potential Solutions
Some python pkg support the interactive with apptainer/singulariy like singularity python. If we use their API to replace the docker pkg, we may be able to use apptainer to maintain the exec environment.