search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

job-exec: support configurable prolog/epilog #2205

Open grondo opened 5 years ago

grondo commented 5 years ago

Support for execution of a separate instance job epilog still needs to be added to the job-exec bulk execution module. This is probably not useful for single-user instances (except for testing I suppose), but for a system instance this will be required (and will likely be just a bulk exec of flux-imp epilog JOBID)

grondo commented 3 years ago

From #3155

Olaf Faaland wants to make use of prologue and epilogue scripts on Elmerfudd to:

(1) Run a script to clean up /dev/shm after a job, so that a user who writes data there doesn't reduce the > > amount available to the next user. (2) Drop caches after a job, e.g. echo 3 >/proc/sys/vm/drop_caches My interest in this is primarily to ensure that data and metadata written to a remote file system such as Lustre is flushed to disk before the node is made available to other users. This is partially so that we find out about a problem as early as possible and minimize damage done, and partially so that one user can't hurt the following user's performance. (3) Run a script to set up and destroy a local ephemeral file systems, for use by the user. One example is connecting to remote NVME via nvme-over-fabrics, formatting the connected device with a file system such as xfs, and setting permissions so that the user can write to it; and then un-doing that after the job is complete. (4) Run a script to set up and destroy a shared ephemeral file systems, for use by the user. Another example is setting up and destroying a shared GFS2 file system. Unlike the local file system setup/destroy case, this would likely need to know the set of nodes participating in the job.

These use cases could all be (perhaps suboptimally) solved with the prolog and epilog support we have now with flux-imp run, but this issue of actually running the prolog/epilog will need to be solved first. Our initial plan was to enable prolog/epilog support at the time of the exec system rewrite (#3346), but it sounds like this is higher priority:

@ofaaland also stated in that issue:

We could perhaps work around the "job-shell plugin runs as a user" issue with some creating sudo.d and scripting, but I wonder if prolog/epilog support isn't important for other testing.

Addressing those use cases somehow is definitely required for us to use elmerfudd with flux.

Probably it will not be too much work to run imp run prolog before a job and imp run epilog after if the instance is configured to do so.

I'll bump the priority of this and work on it asap.

grondo commented 2 years ago

@ofaaland - while this specific issue was not addressed, we did add stopgap support for prolog/epilog into the latest flux-core release (v0.31.0).

Currently, only a "job manager" prolog/epilog is supported, which runs on rank 0 before and after each job. However, a program is provided to run a prolog or epilog across the ranks (nodes) which are assigned to the job, thus effectively providing traditional prolog/epilog support. For more information see the Admin Guide's Job prolog/epilog section.

You might be the first user of this feature, so I'm happy to take feedback and help in any way to make sure you can accomplish what you need!

ofaaland commented 2 years ago

That looks great @grondo , thank you. I'll give it a spin!