CDCgov / cfa_azure

Apache License 2.0
8 stars 2 forks source link

`client.use_pool` does not override mount configuration #91

Open arik-shurygin opened 3 weeks ago

arik-shurygin commented 3 weeks ago

in short if you create a pool and mount specific input and output blobs to the node, that mount configuration is locked in place. If in the future you decide to mount new blobs as input and output the mount configuration of that pool will not change, you must create a new pool with a new name that mounts the different storage blobs.

For an example see: https://portal.azure.com/#view/Microsoft_Azure_Batch/PoolMenuBlade/~/mount-configuration/batchAccountId/%2Fsubscriptions%2Fef340bd6-2809-4635-b18b-7e6583a8803b%2FresourceGroups%2FEXT-EDAV-CFA-PRD%2Fproviders%2FMicrosoft.Batch%2FbatchAccounts%2Fcfaprdba/poolId/scenarios_8cpu_pool

This pool was created by the following commands:

client.set_input_container("scenarios-test-container")
client.set_output_container("example-output-scenarios-mechanistic", "output")
client.set_scaling(
    mode="autoscale",
    autoscale_formula_path="secrets/autoscale.txt",
    timeout=TIMEOUT_MINS,
)
client.create_pool(pool_name="scenarios_8cpu_pool")

If I try to rerun this script for launching jobs but instead run the following:

client.set_input_container("scenarios-mechanistic-input", "input")
client.set_output_container("scenarios-mechanistic-output", "output")
client.set_scaling(
    mode="autoscale",
    autoscale_formula_path="secrets/autoscale.txt",
    timeout=TIMEOUT_MINS,
)
client.use_pool(pool_name="scenarios_8cpu_pool")

the new input and output containers will not be mounted to nodes within scenarios_8cpu_pool instead, the old ones will be mounted from when the pool was created.

I am unsure if this is expected behavior but this should definitely be documented if so, as there are no warnings that the new input/output containers are not being attached.

ryanraaschCDC commented 2 weeks ago

This is intended from the way Azure works. However, it might be beneficial to create an update_pool_settings() method to account for changes you wish to make to an existing pool. I'll look into how we can update pool settings and I will add a warning in the use_pool() method.