Closed sverhoeven closed 8 months ago
The sftp filesystem class copies the zip contents to remote machine, then the mv * output/
is run remotely and lastly the whole job dir is copied back retaining any files that where there before.
The importrun application should be run on a destination with a local filesystem.
You should supply a custom destination picker which directs importrun jobs correctly.
In #90 an EagerScheduler and pickbyname picker where added.
So now you can have the following config:
applications:
calc:
command_template: <calculate something>
move:
command_template: mkdir -p output && mv * output || true
destination_picker: bartender.picker:pick_byname
destinations:
calc:
scheduler:
type: slurm
ssh_config:
hostname: slurm
username: xenon
password: javagat
filesystem:
type: sftp
ssh_config:
hostname: slurm
username: xenon
password: javagat
entry: /home/xenon
move:
scheduler:
type: eager
filesystem:
type: local
The calc app will be executed by a remote slurm and the move command will be executed in the bartender submit request handler.
When using a remote filesystem like sftp the
importrun
application duplicates the zip contents.The job dir looks like
I expected it to no longer have
<zip contents>
in root.