Override snakemake benchmark timer for GPU information

[SamStudio8]

We can spy on the GPU by essentially replicating the way psutil is used for general process monitoring with GPUutil.

[SamStudio8]

I've made a basic implementation towards this by downloading a copy of the snakemake.benchmark module and adding a few GPUtil calls. Annoyingly, the benchmark stuff is pulled in during the executor run-time which means that importing our overridden version in the shell is not enough.

[SamStudio8]

Turns out, you can actually globally override parts of a module by importing it, modifying it, then writing it back to sys.modules. As a duplicated import just returns a link to the module in sys.modules, we can do this in the onstart to hijack the future import in the executor. Neat!

[SamStudio8]

It works! :chefkiss:

[SamStudio8]

max load and mem isn't that informative, so we can do some averages or something next

[SamStudio8]

why have one data point when you can dump them in real time?

[SamStudio8]

this is a little hacky as the benchmark path is not provided to the shell context, but we can just provide a path to a suitable file as part of a job's output settings - it's not perfect but it works. i'm happy with this set-up for capturing GPU information -closing.