(aws-ecs) FargateTaskDefinition EnvironmentFile.fromBucket fails

[robbie1douglas]

Description

When creating a Fargate Task Definition with CDK the task pulling an env file from S3 fails with this error "ResourceInitializationError: failed to download env files: file download command: non empty error stream: RequestCanceled: request context canceled caused by: context deadline exceeded" but what appears to be the exact same configuration created in the console by hand works.

What I'm trying

import * as ec2 from '@aws-cdk/aws-ec2';
import * as ecr from '@aws-cdk/aws-ecr';
import * as ecs from '@aws-cdk/aws-ecs';
import * as ecs_patterns from '@aws-cdk/aws-ecs-patterns';
import * as s3 from '@aws-cdk/aws-s3';
import * as cdk from '@aws-cdk/core';
import * as iam from '@aws-cdk/aws-iam';

export class EcsFargateService extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = ec2.Vpc.fromLookup(this, 'DefaultVPC',{isDefault: true});

    const cluster = new ecs.Cluster(this, "docker-demo", {
      vpc: vpc
    });

    const ecRepo = ecr.Repository.fromRepositoryName(this, 'ecRepo', 'docker-demo');

    const iamRole = iam.Role.fromRoleArn(
      this,
      'IamRole',
      'arn:aws:iam::##########:role/ecsTaskExecutionRole'
    );

    const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {
      executionRole: iamRole,
      taskRole: iamRole
    });

    const s3Bucket = s3.Bucket.fromBucketName(this, 's3Bucket', 'cdk-docker-demo');

    taskDefinition.addContainer('DefaultContainer', {
      image: ecs.ContainerImage.fromEcrRepository(ecRepo),
      memoryLimitMiB: 256,
      environmentFiles: [
        ecs.EnvironmentFile.fromBucket(s3Bucket, 'demo-env-file.env'),
      ],

    });

    const ecsService = new ecs.FargateService(this, 'Service', {
      cluster,
      taskDefinition,
      platformVersion: ecs.FargatePlatformVersion.VERSION1_4
    });
  }
}

IAM (that works in the console just fine)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
            ],
            "Resource": "arn:aws:ecr:us-east-1:#######:repository/docker-demo",
            "Effect": "Allow"
        },
        {
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Environment

• CDK CLI Version : 1.84.0
• Framework Version: ????
• Node.js Version: v12.19.0
• OS : MacOS 10.15.7
• Language (Version): TypeScript

Other

Maybe related: https://stackoverflow.com/questions/65807337/fargate-containers-intermittently-fail-due-to-s3-environment-files-timeout/65913027#65913027

---

This is :bug: Bug Report

[carlosfvp]

I'm having the same issue at the time but from the Web Console. My task execution role has policies to read the s3 bucket and .env files. Then Fargate task is stopping with this message:

Stopped reason ResourceInitializationError: failed to download env files: file download command: non empty error stream: RequestCanceled: request context canceled caused by: context deadline exceeded

Also, I make sure my subnet can assign public IPs. The fargate task has indeed a public IP assigned while it's running.

[tungqualee]

I have the same issue

[banjerluke]

Same here. Whether creating the service from the web console or the CLI, this error prevents tasks from starting up:

ResourceInitializationError: failed to download env files: file download command: non empty error stream: RequestCanceled: request context canceled caused by: context deadline exceeded

Steps I've taken:

• Added S3 permissions to ecsTaskExecutionRole as in documentation.
• Ensured env file is accessible through that role through the IAM simulator.
• Ensured I'm using Fargate platform 1.4.0. (This is a new project I'm setting up.)
• Enabling public IP assignment for the container (was disabled).
• Banged my head against the nearest wall.

None of the steps had any effect beyond a headache.

[travisnguyen20]

@banjerluke Ensure that you put the service/container in a public subnet. In my case, I forgot to attach the internet gateway to the route table so my container can't connect to the internet.

[banjerluke]

Ensure that you put the service/container in a public subnet. In my case, I forgot to attach the internet gateway to the route table so my container can't connect to the internet.

In my case it was a different issue - linking the wrong security group to the service - but still due to a mistake on my end, not a bug. Nevermind! And thanks. 😊

[github-actions[bot]]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[nam169]

facing the same issue

[HappyBoy19940506]

I solved this issue by following the steps below, hope that helps.

Error: "ResourceInitializationError: failed to download env files: file download command: non-empty error stream: RequestCanceled: request context cancelled caused by: context deadline exceeded

Reason: This may be because the subnets that hold your Farget tasks do not have an accessible way to visit your S3 bucket or your SSM parameter store via HTTP(s) request. There are three ways to let Farget visit your other AWS resources like S3.

1. IF you put Farget in public subnets. The public subnets must be associated with an Internet Gateway and enable auto assign Public IPs.
2. IF you put Farget in private subnets. The public subnets must be associated with a Nat Gateway.
3. Or you can use the VPC endpoint to reach other AWS resources. In this way, no associated Gateway is needed.

---

Error : ResourceInitializationError: failed to download env files: file download command: non-empty error stream: failed to download file 1.env: failed to write to a temporary file: AccessDenied: Access Denied

If you get this AccessDenied: Access Denied error.

Reason: This is usually due to permission problems. Go Check the Required IAM permissions part to modify the ecsTaskExecutionRole.

Other weird things: As the doc said, You need to attach an inline_policie to the ecsTaskExecutionRole IAM rather than create a specific another Task _Role for your tasks, which means task_role, as well as task_execution_role, meant to be exactly the same one. But if I separate them apart, like Task Role for only s3 read policy, Task execution role for only default AmazonECSTaskExecutionRolePolicy. It will lead to Access Denied problem. Not quite sure if it's a bug or if I misunderstand something. Apparently, the task role is designed for passing env into a container. But not only permission to visit s3/SSM, but the original AmazonECSTaskExecutionRolePolicy is also needed.

---

facing the same issue

[MrArnoldPalmer]

Since users are still experiencing this, and it seems like it's not immediately obvious what is required to get this to work, I'm going to reopen this. It seems worth investigating whether or not we should include logic to make sure that subnets and iam permissions are correct when calling EnvironmentFile.fromBucket for task defs, IE automatically setting the correct subnet access rules.

[pinarkaymaz6]

is there any update on this? EnvironmentFile.fromBucket still buggy.

[chnacib]

Its a network issue. If your subnet is using internet gateway in route table, check if public ip is enabled on ECS. If your subnet is using NAT Gateway in route table, you can disable public ip, but check if the internet is reachable(in case of not reachable, check if your NAT Gateway is deployed in a public subnet)

[obinnaaliogor]

This really helped me. in my case i entered "tcp" as protocol for the egress rule instead of -1 egress { from_port = 0 to_port = 0 protocol = -1 cidr_blocks = ["0.0.0.0/0"] }

[kumarabhirup]

This really helped me. in my case i entered "tcp" as protocol for the egress rule instead of -1 egress { from_port = 0 to_port = 0 protocol = -1 cidr_blocks = ["0.0.0.0/0"] }

Tried that too, facing the same issue.

[chnacib]

@kumarabhirup check if your private subnet can reach the internet or S3 private endpoint. If using a public subnet, must enable public ip on ECS service