search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
162 stars 49 forks source link

add command to replicate a directory tree on all broker ranks #4807

Open garlick opened 1 year ago

garlick commented 1 year ago

Problem: using the proposed file broadcast toolset in #4789, it is somewhat error prone to set up replication of a directory to identical /tmp paths across an instance.

Example: replicate flux-core source tree across an instance

$ git clone https://github.com/flux-framework/flux-core.git /tmp/flux
$ flux filemap map -C /tmp/ flux
$ flux exec -x 0 flux filemap get -C /tmp
$ flux filemap unmap

It's too easy to get this wrong. Missing the -x 0 would extract over the top of the source. Failing to unmap might result in SIGBUS crashes or hash errors if the rank 0 content is modified while it is still mapped. Therefore, it may be useful to provide a single command that takes care of everything for this use case, e.g.

$ flux bcast /tmp/flux

or maybe

$ flux filemap bcast /tmp/flux
grondo commented 1 year ago

It is appealing to use flux bcast since that would allow this command to be written in Python or the shell, making some of possible boilerplate a little less onerous. It would also make the command more accessible to extensions/enhancements/improvements if necessary.