search

princeton-nlp / intercode

[NeurIPS 2023 D&B] Code repository for InterCode benchmark https://arxiv.org/abs/2306.14898
https://intercode-benchmark.github.io/
MIT License
194 stars 32 forks source link

Running without Docker #3

Open evgenii-nikishin opened 1 year ago

evgenii-nikishin commented 1 year ago

Hi Authors,

Thanks for building this environment, that's a really great contribution.

I was wondering if it's possible to extend the codebase and either get rid of the dependency on Docker or make it compatible with other container technologies (such as Apptainer)? The reason I'm asking is that Docker is not available on the Canadian cluster (and probably on some other clusters too) because of its security risks (https://docs.alliancecan.ca/wiki/Apptainer#Other_Linux_container_technologies).

Thank you.

john-b-yang commented 1 year ago

Hi @evgenii-nikishin thanks for your comment and kind words! We really appreciate it :)

Thanks for pointing out this concern, that is a good point and stringent barrier of entry that our team has not considered so far.

I think between the two options you presented, my guess is that making it compatible with other container technologies (i.e. Apptainer) would likely require less engineering. I have put this down as a to-do and will be sure to look into it, but I think it will take me some time to deliver on this (~1 month).

On the other hand, it seems to me that removing the docker dependency and containerized execution in general means that you'd be interested in running InterCode directly on a machine's a code interpreter or executable? I think this would be possible, but as a word of caution, it may subject the machine to LLM code generations that might be cause irreversible modifications to the host system (i.e. rm -rf ...).

Between these two choices, do you have a preference for one or the other? I personally think the first option (compatibility with other container tech.) is better, but I'll prioritize whatever you prefer!

evgenii-nikishin commented 1 year ago

Thank you for the quick response! I think making it compatible with Apptainer should be a better alternative.

As for removing the containerized execution altogether, I wouldn't want that as indeed it might lead to unsafe behavior; I was just thinking that maybe there are other ways to create a sandbox without docker.

john-b-yang commented 1 year ago

Gotcha, sounds good! I'll give it a shot and see how things go - will do my best to make it happen! I'll leave this open and ping here when I have an update.