Unable to push using kaniko

[shreera1]

Hi @ollypom I went over the very detailed and helpful tutorial https://aws.amazon.com/blogs/containers/building-container-images-on-amazon-ecs-on-aws-fargate/, but I am unable to push a docker image present on a private github repo using kaniko present in an ECR repository. However, I was able to push if I pass AWS_ACCESS_KEY_ID, AWS_SESSION_TOKEN and AWS_SECRET_ACCESS_KEY as environment variables. I want to remove those env variables in the ecs task definition file and successfully push. I am getting the following error: 401 Unuathorized. Any help will be appreciated.Thanks

[ollypom]

Hey @shreera1 , which version of Kaniko are you using? There was an issue in v1.19.0 and 1.19.1 that prevented Kaniko from pushing to ECR but that was fixed in 1.19.2. https://github.com/GoogleContainerTools/kaniko/issues/2882

This Task Definition has just worked ok for me:

{
    "family": "kaniko-builder",
    "taskRoleArn": "arn:aws:iam::111222333444:role/Kaniko_Task_Role",
    "executionRoleArn": "arn:aws:iam::111222333444:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "containerDefinitions": [
        {
            "name": "kaniko",
            "image": "gcr.io/kaniko-project/executor:v1.19.2",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/aws/ecs/service/kaniko",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "kaniko"
                }
            },
            "command": [
                "--context",
                "git://github.com/ollypom/mysfits.git",
                "--context-sub-path",
                "./api",
                "--dockerfile",
                "Dockerfile.v3",
                "--destination",
                "111222333444.dkr.ecr.eu-west-1.amazonaws.com/mysfits:latest",
                "--force"
            ]
        }
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024"
}

Note the taskRoleArn here. By adding this role into the Task you are injecting AWS credentials into the container so you do not have to use the AWS_ environment variables. My role has 1 policy attached: arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser.

[shreera1]

Hi @ollypom, Thanks for your reply. Apologies, my question was not well phrased. I am using an IAM role, specifically a role that is developed for github actions (https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/). I am also configuring aws-credentials using the following

 - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::123456789100:role/my-github-actions-role
        aws-region: us-east-2

So, my CI image does not have the aws credentials file that is required for kaniko to authenticate. Do you have any suggestions how I can avoid passing AWS_ credentials as environment variables?

[ollypom]

Interesting. Are you using Fargate to run Kaniko or are you instead running Kaniko on a GH (self)hosted Runner as the next step in your pipeline?

[shreera1]

I am using FARGATE to run kaniko..

[ollypom]

Ah ok, sorry needed to clarify. I feel like I'm missing something.

Are you using the GH action to create / register the Task Definition (passing in the environment variables) or are you using the GH action to run a task (passing in variables as a container override)?

[shreera1]

Aah. My bad, let me share the complete flow so maybe you can have a better idea.

1. I have a github self-hosted runner which is launched via a ECS service and it is running on FARGATE.
2. Now, I am using kaniko and defining all the resources such as clusters, task definition, etc (Example task definition below).
3. As you see, I am using terraform to create the resources, Now in my github actions, after configuring aws credentials (using the action), I run terraform apply with temporary AWS credentials generated by the action and pass them as arguments. That step will create all the necessary resources on AWS. The next step is I run an aws ecs run-task with the necessary arguments on the github runner.

resource "aws_ecs_task_definition" "kaniko-demo" {
    family                   = "kaniko-demo"
    network_mode             = "awsvpc"
    requires_compatibilities = ["FARGATE"]
    cpu                      = 2048
    memory                   = 16384
    execution_role_arn       = aws_iam_role.ecsTaskExecutionRole-kaniko-demo.arn
    task_role_arn            = aws_iam_role.kaniko-demo_ecs_role.arn

    container_definitions = <<DEFINITION
    [
          {
              "name": "kaniko",
              "image": "1234567890.dkr.ecr.us-east-1.amazonaws.com/kaniko:latest",
              "networkMode": "awsvpc",
              "cpu": 2048,
              "memory": 16384,
               "logConfiguration": {
                  "logDriver": "awslogs",
                  "options": {
                      "awslogs-group": "logs-kaniko-demo",
                      "awslogs-region": "us-east-1",
                      "awslogs-stream-prefix": "kaniko-demo"
                  }
              },
            "environment": [
                {
                    "name": "AWS_ACCESS_KEY_ID",
                    "value": "${var.access_key}"
                },
                {
                    "name": "AWS_SECRET_ACCESS_KEY",
                    "value": "${var.secret_access_key}"
                },
                {
                    "name": "AWS_SESSION_TOKEN",
                    "value": "${var.session_token}"
                }
          ],
              "command": [
                  "--context",
                  "git://mygithub-repo.git#refs/heads/main",
                  "--context-sub-path", 
                  "./api",
                  "--dockerfile", 
                  "Dockerfile",
                  "--destination", 
                  "1234567890.dkr.ecr.us-east-1.amazonaws.com/kaniko-demo:latest",
                  "--force",
                  "--verbosity", "trace"
                  ]
          }
    ]
  DEFINITION

  }