Feedback on user documentation

[rwf14f]

Just a few notes on the user documentation:

Wallclock Time:

• There's a maximum wallclock time configured. It's currently 72 hours, jobs get killed if they exceed this.
• The same is true for CPU time for each requested CPU (ie job gets killed if total used CPU hours > requested CPUs * 72 hours)
• Need to check how the machine count fits into this, we'll most likely have to update the configuration to take this into account (ie machine count requested CPUs 72 hours).

Container support:

• The docker universe won't work because there's no docker on the worker nodes.
• The worker nodes have apptainer installed though. Jobs can use it to run their workloads inside a container.
• Running containers in a vanilla universe job works fine.
• No experience with doing the same in parallel universe jobs. There might be communication problems between the mpi processes if mpi is running inside a container.

[rwf14f]

Condor now also has a container universe which can run jobs in an apptainer container, so doesn't need docker installed.

[ickc]

About wall clock time, when is the 2nd constraint not the same as the first constraint? How is it going to track CPU hours?

[ickc]

We will not document the use of container above as it is not a supported configuration from us (SO:UK Data Centre) where we only tell the users "welcome to try". The notes above probably would be useful, but as far as I understand, HTCondor's universes are mutually exclusive as explained in their documentation, so probably parallel universe won't work with container universe. Unless this can be done, we should discourage the users to use container as that would limited their workflows to single node usage.

The wallclock time needs to be documented, but I don't think we have all the information to add such page yet. I envisioned it to be something like https://docs.nersc.gov/jobs/policy/ where wall-clock time should be part of the constraints on the computer systems that the user should be aware of. C.f. #6.

Pushing back to next release.

[ickc]

Addressed by fe63759, 71aa29e, and continue to #35 for the remaining clarification needed.