Closed turian closed 3 years ago
Alternately, I would like to have files 'local.py.template1', 'local.py.template2', etc. and specify the file for each instance, and it becomes 'local.py'
Each container has a name. By default, it's default
, but you can change it using the name
parameter (see here). This name is used to assign the container to a specific instance. By default, the container with the name default
will be assigned to all your instances, but it can be changed using the containerName
parameter in the instance configuration (see here).
So, in your case, I would create several containers with different environmental variables as you suggested. Then, based on the environmental variable, your script could pick up correct hyperparameters for the model. To reduce the amount of copy-pasting, you can use YAML aliases. Something like that:
containers:
- &DEFAULT_CONTAINER
projectDir: /workspace/project
file: docker/Dockerfile.spotty.gpu
volumeMounts:
- name: project
mountPath: /workspace/project
env: &COMMON_ENVS
PYTHONPATH: /workspace/project
# ...
- <<: *DEFAULT_CONTAINER
name: model-1
env:
<<: *COMMON_ENVS
MODEL_NAME: model1
- <<: *DEFAULT_CONTAINER
name: model-2
env:
<<: *COMMON_ENVS
MODEL_NAME: model2
instances:
- name: aws-1
provider: aws
parameters: &INSTANCE_PARAMS
containerName: model-1
region: eu-central-1
instanceType: p3.2xlarge
# ...
- name: aws-2
provider: aws
parameters:
<<: *INSTANCE_PARAMS
containerName: model-2
I am hoping to spin up a fleet of spotty instances, each identical except for having one variable (hyperparameter set name) different for each.
Something like: HYPERPARAMETERS = ["model1", "model2", etc.] with each instance having a different value.
I guess an environment variable makes the most sense. I see the ability to add envs to the containers. But I don't find the documentation on what happens if you have multiple containers:
One workaround would be to have a separate spotty.yaml for each instance (and hyperparameter set name) but that seems janky.