search

spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
493 stars 43 forks source link

Running multiple instances #27

Closed maclandrol closed 5 years ago

maclandrol commented 5 years ago

Hello @apls777

Thanks for spotty, I love this project. I was wondering if there is way to request several instances at the same time (like with sagemaker) to run the same code but with different configuration in parallel.

Also is it possible to pass arguments to spotty run like `spotty run train --config "path to file"``

Thanks.

apls777 commented 5 years ago

Hi @maclandrol,

Thank you! :)

All spotty commands have the -c/--config parameter. So you can start several instances by providing different spotty configuration files to the spotty start command, and then run your script on each instance providing the same configuration file to the spotty run command. For example:

spotty start -c spotty.i1.yaml
spotty run train -c spotty.i1.yaml

spotty start -c spotty.i2.yaml
spotty run train -c spotty.i2.yaml

And don't forget to stop all of them after :).

I know that it's not very convenient, because you're basically copy-pasting the same configuration file several times, and then using the -c parameter for every command. Moreover, the project name parameter should be different in each configuration file, otherwise, the tool tells you that the stack already exists. Also, it will create a new S3 bucket for each configuration.

I faced this problem before as well, and I've made it slightly better it in the future version 1.2. The format of the configuration file will be a little bit different, and you will be able to specify there several instances. They will reuse the same S3 bucket. Also, scripts will be parameterizable, so you could use custom parameters: spotty run [INSTANCE_NAME] SCRIPT_NAME [-p PARAM1=VAL1 PARAM2=VAL2 ...].

maclandrol commented 5 years ago

This answers all my question. Thanks !

I am using spotty inside a wrapper for automatic config generation and instance launching, so it should be fine.

Reusing the same S3 bucket will be a really great feature. It might also be helpful to have a way to automatically stop the instance after a given time limit (I am using a scheduler now, which is working fine) !

apls777 commented 5 years ago

Thank you for the feedback! I will add the time limit feature to my TODO list.