dstackai / dstack

dstack is an open-source alternative to Kubernetes, designed to simplify development, training, and deployment of AI across any cloud or on-prem. It supports NVIDIA, AMD, and TPU.
https://dstack.ai/docs
Mozilla Public License 2.0
1.48k stars 152 forks source link

Can't use spot instances if `AWSServiceRoleForEC2Spot` is not created in the AWS account #886

Closed peterschmidt85 closed 3 weeks ago

peterschmidt85 commented 9 months ago

To use spot instances with an AWS account, the account needs to have the corresponding service-linked role created (AWSServiceRoleForEC2Spot). If this role is not created, AWS will throw the following error upon an attempt to run a spot instance:

WARNING 2024-02-08T17:42:46.589 dstack._internal.core.backends.aws.compute Got botocore.exceptions.ClientError: An error occurred (AuthFailure.ServiceLinkedRoleCreationNotPermitted) when calling the RunInstances operation: The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances.

From the docs:

Under most circumstances, you don't need to manually create a service-linked role. Amazon EC2 creates the AWSServiceRoleForEC2Spot service-linked role the first time you request a Spot Instance using the console.

Solution

If the role isn't created (e.g. if the AWS account was created long time ago), the role has to be created manually by the following command:

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Note, this is not dstack's issue. If a spot instance cannot be created because of the lack of this role, dstack treats this as spot instances not available.

peterschmidt85 commented 6 months ago

This issue is stale because it has been open for 30 days with no activity.

peterschmidt85 commented 5 months ago

This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 3 weeks ago

This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.