search

WIPACrepo / iceprod

IceCube dataset management system
MIT License
4 stars 4 forks source link

singularity container support #352

Closed dsschult closed 3 weeks ago

dsschult commented 1 year ago

Jobs are starting to use containers, so we should support this with explicit syntax.

dsschult commented 1 year ago

To properly support this, singularity should run at the task level, as part of the HTCondor job. This means not running IceProd code on the worker, or only a very limited amount that can run in any environment.

Plans:

  1. Convert file transfers to HTCondor file transfers, possibly with a custom plugin.
  2. Convert the entire task to either a bash or simple python script, with iterations rolled out.

This will be a major release.