hpc / charliecloud

Now hosted on GitLab.
https://gitlab.com/charliecloud/main
Apache License 2.0
313 stars 61 forks source link

`ch-fromhost`: update or replace #1861

Open reidpr opened 6 months ago

reidpr commented 6 months ago

ch-fromhost has a complicated and messy task, and this is made more difficult by some drawbacks that have both been designed in and accumulated over time:

  1. The script is 500 lines of fairly dense POSIX sh (not even Bash).

  2. It does not support r/o images, e.g. SquashFS which are important for scaling (#286).

  3. Lack of standardization. Vendors are starting to provide OCI hooks that do a similar task, and we can’t use them.

  4. Adds a separate step to the workflow (#1361).

  5. For nVidia, uses an external program (nvidia-container-cli) that must be installed and kept up to date (#940).

This issue is to make this feature easier to maintain. Options I can think of are:

  1. Rewrite the script in Python.
  2. Drop the script in favor of OCI hooks (whether done directly by the ch-run or a Python helper).
  3. Do nothing.

I marked the priority “high” because I think this may be important for supporting AI workloads well.