pilot file staging

[dsschult]

Based on the gpu studies, it's clear that we spend a significant fraction of time staging files in or out. One idea is to let the pilot do this work before and after a job runs, so more time can be spent on the gpu. This would require smart downloading of a config file early (somehow knowing when a job is about to vacate, or double scheduling jobs for certain resources).

2 possible approaches:

1) Hoist some of the config parsing up to the pilot so it can get the files to download/upload.

• This is basically redoing most of what happens in i3exec and exe.

2) Establish pre, run, post phases of task running, with communication to the pilot.

• Inter-process communication

[dsschult]

With the terrible networking performance at some sites, we're losing over 50% of the gpu computing time. Raising priority on this issue.