search

spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
492 stars 43 forks source link

Can't use existing IAM role in spotty #42

Closed MiroFurtado closed 3 years ago

MiroFurtado commented 5 years ago

In more managed environments, you don't always have the flexibility to create your own IAM role versus use the one that is assigned to you. Spotty should have the ability to use an already existing IAM role if it has the right permissions.

Opening this as an issue so I can keep track of it, I'll be submitting a pull request next week likely.

apls777 commented 5 years ago

At the moment Spotty creates an IAM role and an Instance Profile that wraps that role to use it with an EC2 instance. I assume that in some environments users also cannot create their own Instance Profile. Maybe then better to add functionality to use an existing Instance Profile?

Let's say it will be the instanceProfileArn parameter, and if it's specified then Spotty will use it instead of creating one. What do you think?

Also, I have a question about your environment: do you have permissions to create your own bucket? Or is it an issue as well? Because Spotty creates a bucket that is being used to sync the project and download files.

tsdalton commented 4 years ago

Hello, I also have issues with using Spotty in a managed AWS environment and would love the ability to provide a predefined IAM Role and Instance Profile as a configuration option. Is this feature in the pipeline and what would you estimate the level of effort required to implement?

To answer your S3 bucket question: yes I can create S3 buckets in my particular environment. In more restrictive environments where wholesale bucket creation is not an option I imagine that admins could grant users permissions to create buckets under some ARN naming restrictions. An admin may allow a user to create a private bucket conforming to some naming convention as long as the bucket is private, for example.

MiroFurtado commented 4 years ago

Apologies for never responding to the above questions, I must have missed this. I did actually end up modifying the spotty code to allow this, but I don't remember what I did and I've since lost the code. The fix wasn't too difficult if I remember correctly.

apls777 commented 4 years ago

@tsdalton Actually, it's straight forward. I cannot promise that I'll release this feature in the following days as I'm a little busy with another project, but maybe I'll get some spare time.

For now, you can just clone the repo and replace the call that creates the Instance Profile stack with your Instance Profile ARN here: https://github.com/apls777/spotty/blob/e67cae8afec893ec32c80a2f158a9fe3566b144c/spotty/providers/aws/deployment/instance_deployment.py#L73

But if you'll also create the instanceProfileArn parameter in the configuration file and will use it when it's specified, feel free to open a PR :).

tsdalton commented 4 years ago

@apls777 Thanks I have indeed updated my fork to surface the instanceProfileArn parameter to the config file so I'll open up a pull request when I get a chance to clean things up and merge-ready, probably towards the end of the week.

tsdalton commented 4 years ago

@apls777 sorry it took me so long, got sidetracked. I've opened a pull request, please review it at your leisure!

apls777 commented 3 years ago

Thank you @tsdalton, this functionality now released with Spotty 1.3.0.