search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.81k stars 1.84k forks source link

ECS Run Task does not work if Launch Type is passed as argument. #4160

Closed JPBou closed 2 weeks ago

JPBou commented 2 weeks ago

Describe the bug

I have a task definition that runs on ECS and uses EC2 as its launch type.

When calling this, the task is submitted successfully to the cluster, even if there are no running instances (I have my ASG taking care of that):

def_dict = {
"cluster":"ecs-gpu-cluster",
"taskDefinition":"arn:aws:ecs:us-east-2:058442094236:task-definition/prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d:8",
}
ecs_client.run_task(**def_dict)

Response:

{'tasks': [{'attachments': [],
   'capacityProviderName': 'ec2-gpu-asg-provider',
   'clusterArn': 'arn:aws:ecs:us-east-2:058442094236:cluster/ecs-gpu-cluster',
   'containers': [],
   'cpu': '256',
   'createdAt': datetime.datetime(2024, 6, 10, 16, 59, 36, 935000, tzinfo=tzlocal()),
   'desiredStatus': 'RUNNING',
   'enableExecuteCommand': False,
   'group': 'family:prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d',
   'lastStatus': 'PROVISIONING',
   'launchType': 'EC2',
   'memory': '512',
   'overrides': {'containerOverrides': [],
    'inferenceAcceleratorOverrides': []},
   'tags': [],
   'taskArn': 'arn:aws:ecs:us-east-2:058442094236:task/ecs-gpu-cluster/566375b43d9c492ca5e2db8f5d78dcc0',
   'taskDefinitionArn': 'arn:aws:ecs:us-east-2:058442094236:task-definition/prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d:8',
   'version': 1}],
 'failures': [],
 'ResponseMetadata': {'RequestId': 'df57061a-6074-4783-a41e-ebccbb460c16',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amzn-requestid': 'df57061a-6074-4783-a41e-ebccbb460c16',
   'content-type': 'application/x-amz-json-1.1',
   'content-length': '742',
   'date': 'Mon, 10 Jun 2024 19:59:36 GMT'},
  'RetryAttempts': 0}}

But if I submit my task passing launch type as a parameter, the call will error out when there are no instances running on the cluster:

def_dict = {
"cluster":"ecs-gpu-cluster",
"launchType":"EC2",
"taskDefinition":"arn:aws:ecs:us-east-2:058442094236:task-definition/prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d:8",
}
ecs_client.run_task(**def_dict)

Error log:

---------------------------------------------------------------------------
InvalidParameterException                 Traceback (most recent call last)
Cell In[32], [line 6](vscode-notebook-cell:?execution_count=32&line=6)
      [1](vscode-notebook-cell:?execution_count=32&line=1) def_dict = {
      [2](vscode-notebook-cell:?execution_count=32&line=2) "cluster":"ecs-gpu-cluster",
      [3](vscode-notebook-cell:?execution_count=32&line=3) "launchType":"EC2",
      [4](vscode-notebook-cell:?execution_count=32&line=4) "taskDefinition":"arn:aws:ecs:us-east-2:058442094236:task-definition/prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d:8",
      [5](vscode-notebook-cell:?execution_count=32&line=5) }
----> [6](vscode-notebook-cell:?execution_count=32&line=6) ecs_client.run_task(**def_dict)

File ~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:565, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    [561](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:561)     raise TypeError(
    [562](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:562)         f"{py_operation_name}() only accepts keyword arguments."
    [563](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:563)     )
    [564](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:564) # The "self" in this scope is referring to the BaseClient.
--> [565](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:565) return self._make_api_call(operation_name, kwargs)

File ~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1021, in BaseClient._make_api_call(self, operation_name, api_params)
   [1017](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1017)     error_code = error_info.get("QueryErrorCode") or error_info.get(
   [1018](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1018)         "Code"
   [1019](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1019)     )
   [1020](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1020)     error_class = self.exceptions.from_code(error_code)
-> [1021](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1021)     raise error_class(parsed_response, operation_name)
   [1022](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1022) else:
   [1023](https://file+.vscode-resource.vscode-cdn.net/Users/joaopedroboufleur/Downloads/~/Library/Caches/pypoetry/virtualenvs/lvc-QvPBVAZ5-py3.10/lib/python3.10/site-packages/botocore/client.py:1023)     return parsed_response

InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: No Container Instances were found in your cluster.

Expected Behavior

The API call should for run task should work when submitting a task to a cluster with no instances when I pass in launch type as a parameter.

Current Behavior

The run task API does not work when submitting launch type as a parameter if the cluster has no running instances.

Reproduction Steps

ecs_client = boto3.client('ecs') def_dict = { "cluster":"ecs-gpu-cluster", "launchType":"EC2", "taskDefinition":"arn:aws:ecs:us-east-2:058442094236:task-definition/prefect_ecs-gpu-pool_47cd3a8c-d732-4670-83e5-1d12e2f43d6d:8", } ecs_client.run_task(**def_dict)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.122

Environment details (OS name and version, etc.)

Linux

DbCrWk commented 2 weeks ago

I'm having the exact same issue.

tim-finnigan commented 2 weeks ago

Thanks for reaching out. The ECS run_task command involves the underlying RunTask API, so the issue here relates to the ECS service rather than Boto3 directly. I found this other issue referencing the same error message: https://github.com/boto/boto3/issues/2862. The creator of that issue closed it and noted that "passing launchType overrides the capacity provider settings, thus ignoring the autoscaler." So it seems like omitting launchType is required here, but that isn't fully clear in the documentation or error message.

I will reach out to the ECS team for clarification, and created https://github.com/aws/aws-sdk/issues/761 to track this in our cross-SDK repository going forward. So please refer to that issue for updates. And for documentation feedback in the future you can also use the Provide feedback at the bottom of the API documentation page to send feedback directly, and any updates to those docs would automatically get reflected in the Boto3 docs.

github-actions[bot] commented 2 weeks ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.