[ECS] [request] running an ecs task should allow awslogs-stream-prefix being overridden

[lasergoat]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

When using code like this python script using boto3.client('ecs').run_task(), I should be able to specify a awslogs-stream-prefix for the awslogs driver within the container overrides. This way, I can clearly see which log stream corresponds to which fargate ecs task. Here's an example of using run_task():

result = ecs_client.run_task(
    cluster=clusterName,
    taskDefinition=taskDefinitionName,
    launchType='FARGATE',
    networkConfiguration={
        'awsvpcConfiguration': {
            'subnets': [
                ...
            ]
        }
    },
    overrides={
        'containerOverrides': [
            {
                'name': taskDefinitionName,
                // let me specify a log driver override with options like 
                'command': [
                    '/bin/sh',
                    '-c',
                    f'make deploy_qa env=qa-{build_id} args="{arg_string}"',
                ]
            }
        ]
    }
)

Which service(s) is this request for?

ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Right now, when I look at my log streams for this ecs task definition (see screenshot below), I have to find logs either by searching or using the timestamp. However, each log stream within my log group does correspond 1:1 with a build_id - but it's too hard to find which log stream is which, because we have so many of these running in parallel.

When creating a task definition I can specify a stream prefix BUT when running the task, I can specify all sorts of overrides, except for the stream prefix. Here are the docs on specifying the stream prefix during task definition creation. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

Are you currently working around this issue?

yes, currently I attempt to find the right stream by the timestamp which isn't too hard but it's not straightforward.

Additional context

I've opened a support task and they told me it's not possible and to request a community issue. Case id 9419954151

Attachments

Here's a screenshot of my log group. Each stream corresponds to a separate build_id, yet I have no way of getting the build_id to appear in this listing.

[renanwilliam]

Hi @lasergoat

I was in a similar situation, but I used an effective workaround. The pattern for log stream always will be:

• {prefix in task definition}/{container name}/{ecs task id}

What I do is: during the task, I save the ECS Task ID in DynamoDB, relating the task context. So, when I need to get the traces, I have all information required to get the proper streaming log.

Isn't work for you?