i-VRESSE / bartender

Middleware web service to schedule jobs on various infrastructures
https://i-vresse-bartender.readthedocs.io
Apache License 2.0
1 stars 0 forks source link

importrun and remote filesystem gives duplicate #89

Closed sverhoeven closed 8 months ago

sverhoeven commented 8 months ago

When using a remote filesystem like sftp the importrun application duplicates the zip contents.

The job dir looks like

<zip contents>
output/<zip contents>
output/stderr.txt
output/stdout.txt
returncode

I expected it to no longer have <zip contents> in root.

sverhoeven commented 8 months ago

The sftp filesystem class copies the zip contents to remote machine, then the mv * output/ is run remotely and lastly the whole job dir is copied back retaining any files that where there before.

The importrun application should be run on a destination with a local filesystem.

You should supply a custom destination picker which directs importrun jobs correctly.

sverhoeven commented 8 months ago

In #90 an EagerScheduler and pickbyname picker where added.

So now you can have the following config:

applications:
  calc:
    command_template: <calculate something>
  move:
    command_template: mkdir -p output && mv * output || true
destination_picker: bartender.picker:pick_byname
destinations:
  calc:
    scheduler:
      type: slurm
      ssh_config:
        hostname: slurm
        username: xenon
        password: javagat
    filesystem:
      type: sftp
      ssh_config:
        hostname: slurm
        username: xenon
        password: javagat
      entry: /home/xenon
  move:
    scheduler:
      type: eager
    filesystem:
      type: local

The calc app will be executed by a remote slurm and the move command will be executed in the bartender submit request handler.