search

QIICR / SlicerGCPSetup

1 stars 0 forks source link

Instructions to automate creation, initialization and deletion of instances #4

Open fedorov opened 5 years ago

fedorov commented 5 years ago

@pieper: In addition to warning people, we could tell them how to fix the problem (delete the instance if you are finished with it).

@fedorov: Is there a way to specify a script that should be run automatically to configure the instance?

@pieper: Yes: --metadata-from-file startup-script=startup-script.sh

https://cloud.google.com/sdk/gcloud/reference/compute/instances/add-metadata

I'll invite you to a colab notebook where I use it.

fedorov commented 5 years ago

to be continued from https://github.com/QIICR/SlicerGCPSetup/commit/6df8068d43387ec29409325855aa539febc3616d#comments

pieper commented 5 years ago

Note that for stopping the instance I include shutdown -h now at the end of the startup script, but this doesn't delete the instance. The instance may have permission to delete itself, but I haven't tried yet. Or maybe there's an autodelete on shutdown option.

Also as a side note, I found that the default quota is 8 simultaneous IP addresses, so that's the number of VMs you can start at a time. I believe it's easy to increase the quota and also maybe to launch instances without external IPs to get around this.

fedorov commented 5 years ago

I think overall, dynamic allocation, or "delete instance when done" is going to be problematic, considering the need to copy the data to the instance to work with the data, unless they use FileStore, which I don't have experience with, and which apparently is very costly. I don't see how it can be practical without some orchestration layer that will implement granular distribution of data and synchronization back to persistent storage and balancing various sources of costs... I think for now, I will add a note that VMs can be deleted to eliminate the costs associated with VM disk storage, but not go beyond that - it is too advanced topic for me to make suggestions at this point!