Add settings option to supply bash commands

[n-gao]

Issue

Currently, there is no option to execute bash commands before the actual python code gets executed. However, there are scenarios where one would like to execute some bash commands prior. Here is an example scenario: One has a regular slurm cluster where one has the main MongoDB. Additionally, one has access to a second slurm cluster but without MongoDB. In order to store all experiments in the same MongoDB one needs a port forward between these two machines. Other use cases would be to mount file systems via ssh.

Solution

This PR enables one to supply additional bash commands via the settings file which get executed at the beginning and end of an experiment. Here's an example settings file:

SETTINGS = {
    "SETUP_COMMAND": "ssh -f -N -L 27018:main.server:27017 main.server",
    "END_COMMAND": "pkill -f 'ssh -f'",
}

[gasteigerjo]

Sounds like a good idea. I guess this doesn't introduce any additional vulnerabilities since you could also run anything via Python.

However, we shouldn't call these "bash" since the user might actually use a different shell. How about PRE_SHELL, PRE_SCRIPT, PRE_COMMAND, PRE_CMD? Or SETUP_COMMAND and END_COMMAND?

[n-gao]

Agree, I think SETUP_COMMAND and END_COMMAND are good options.