wustl-oncology / cloud-workflows

Infrastructure and tooling required to get genomic workflows running in the cloud
1 stars 7 forks source link

Customizing VM's CPU/RAM instead of setting --machine-type #32

Open Layth17 opened 1 year ago

Layth17 commented 1 year ago

The vanilla approach to launch a VM is as follows:

bash start.sh $GCS_INSTANCE_NAME --server-account $GCS_SERVICE_ACCOUNT --project $GCS_PROJECT --boot-disk-size=50GB --boot-disk-type=pd-standard --machine-type=e2-standard-2

However, sometimes we probably want to use --custom-cpu and/or --custom-memory. Using either of those conflicts with using --machine-type. Currently, --machine-type is always invoked:

gcloud compute instances create $INSTANCE_NAME \
       --project $PROJECT \
       --image-family debian-11 \
       --image-project debian-cloud \
       --zone $ZONE \
       --machine-type=$MACHINE_TYPE \
       --service-account=$SERVER_ACCOUNT --scopes=cloud-platform \
       --network=$NETWORK --subnet=$SUBNET \
       --metadata=cromwell-version=71 \
       --metadata-from-file=startup-script=$SRC_DIR/server_startup.py,cromwell-conf=$CROMWELL_CONF,helpers-sh=$SRC_DIR/helpers.sh,cromwell-service=$SRC_DIR/cromwell.service,workflow-options=$WORKFLOW_OPTIONS,persist-artifacts=$SRC_DIR/../scripts/persist_artifacts.py \
       $@

and if $MACHINE_TYPE is empty, then, prior to the block above, machine type is set to the default MACHINE_TYPE=${MACHINE_TYPE:-"e2-standard-2"} making it impossible to avoid.

Which leads to the following error message (gcloud.compute.instances.create) Invalid value for [--machine-type]: Cannot set both [--machine-type] and [--custom-cpu]/[--custom-memory] for the same instance.

Even though the command used, bash start.sh $GCS_INSTANCE_NAME --server-account $GCS_SERVICE_ACCOUNT --project $GCS_PROJECT --boot-disk-size=200GB --custom-memory 64G --custom-cpu 6, did not use the flag --machine-type

For the most part, this is not critical, but could be nice to be add the ability to customize those parameters.