Closed dchaley closed 3 months ago
Runtime environments (aka google projects) have a bunch of configuration that "never" changes no matter the type of job being run.
For example: network settings, service accounts, allowed regions/zones…
We need to be able to easily configure these without requiring users to manually edit the base job JSON data.
./run-multistep-job.py --network "foo" --subnetwork "bar" --service_account "email"
While this is easiest to implement, it's rather awkward & repetitive. It makes it that much harder to run a simple job.
In this case, the script would inspect the environment for specific variables such as $NETWORK and $SERVICE_ACCOUNT.
$NETWORK
$SERVICE_ACCOUNT
This is cleaner than parameters, but still requires additional environment setup (no pun intended).
Pass a config filename on the command-line, and merge that into the base batch job json.
./run-multistep-job.py --config "config.json"
Then we have, in config.json something like:
config.json
{ "allocationPolicy": { "location": { "allowedLocations": [ "region/us-central1" ] }, "network": { "networkInterfaces": [ { "network": "foo", "subnetwork": "bar", "noExternalIpAddress": true } ] }, "serviceAccount": { "email": "a@b.com" } } }
Note that we're following the batch structure exactly, so that this just merges in. It also means users can parameterize anything they want that Batch supports (even if we don't yet support options for it in our scripts) by just editing the config.
Let's go with option 3: the config file.
Runtime environments (aka google projects) have a bunch of configuration that "never" changes no matter the type of job being run.
For example: network settings, service accounts, allowed regions/zones…
We need to be able to easily configure these without requiring users to manually edit the base job JSON data.
Some options
Option 1: CLI parameters
While this is easiest to implement, it's rather awkward & repetitive. It makes it that much harder to run a simple job.
Option 2: environment variables
In this case, the script would inspect the environment for specific variables such as
$NETWORK
and$SERVICE_ACCOUNT
.This is cleaner than parameters, but still requires additional environment setup (no pun intended).
Option 3: configuration file
Pass a config filename on the command-line, and merge that into the base batch job json.
Then we have, in
config.json
something like:Note that we're following the batch structure exactly, so that this just merges in. It also means users can parameterize anything they want that Batch supports (even if we don't yet support options for it in our scripts) by just editing the config.
Decision
Let's go with option 3: the config file.