Unable to push an image to ECR

[jammymalina]

Hi Packer Docker plugin team, I observed this peculiar issue.

Overview of the Issue

Post-processor docker-push fails when it tries to push an image to the ECR. Both build and login were successful. If I try to push the docker image manually (doing docker push myself) it succeeds.

Reproduction Steps

Run packer build

Plugin and Packer version

From packer version 1.8.3 and plugin version 1.0.8

Simplified Packer Buildfile

packer {
  required_plugins {
    docker = {
      version = "1.0.8"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "jdk" {
  image    = "public.ecr.aws/docker/library/openjdk:20-slim-bullseye"
  commit   = true
  platform = "linux/amd64"

  changes = [...]
}

build {
  name = "jdk_image"
  sources = [
    "source.docker.jdk"
  ]

  provisioner "shell" {
    inline = [...]
  }

  post-processors {
    post-processor "docker-tag" {
      repository = "${var.aws_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com/${var.image_name}"
      tags       = [var.image_tag]
    }

    post-processor "docker-push" {
      ecr_login    = true
      login_server = "https://${var.aws_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com"
    }
  }
}

Operating system and Environment details

macOS Monterey 12.6, Darwin 21.6.0 Darwin Kernel Version 21.6.0, Docker with Colima VM

Log Fragments and crash.log files

==> jdk_image.docker.jdk: Running post-processor:  (type docker-push)
    jdk_image.docker.jdk (docker-push): Creating temporary Docker configuration directory
    jdk_image.docker.jdk (docker-push): Fetching ECR credentials...
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Getting ECR token for account: 123456789 in eu-west-1..
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Found region eu-west-1
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 [INFO] AWS authentication used: "SharedCredentialsProvider"
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Successfully got login for ECR: https://123456789.dkr.ecr.eu-west-1.amazonaws.com
    jdk_image.docker.jdk (docker-push): Logging in...
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 login -u AWS --password-stdin https://123456789.dkr.ecr.eu-west-1.amazonaws.com]
    jdk_image.docker.jdk (docker-push): WARNING! Your password will be stored unencrypted in /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013/config.json.
    jdk_image.docker.jdk (docker-push): Configure a credential helper to remove this warning. See
    jdk_image.docker.jdk (docker-push): https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    jdk_image.docker.jdk (docker-push): Login Succeeded
    jdk_image.docker.jdk (docker-push): Pushing: 123456789.dkr.ecr.eu-west-1.amazonaws.com/push-repo:latest
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 push 123456789.dkr.ecr.eu-west-1.amazonaws.com/repo:latest]
    jdk_image.docker.jdk (docker-push): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    jdk_image.docker.jdk (docker-push): Logging out...
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 logout https://123456789.dkr.ecr.eu-west-1.amazonaws.com]
    jdk_image.docker.jdk (docker-push): Removing login credentials for 123456789.dkr.ecr.eu-west-1.amazonaws.com
    jdk_image.docker.jdk (docker-push): Removing temporary Docker configuration directory
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 error: Bad exit status: 1
2022/10/27 17:23:46 [INFO] (telemetry) ending docker-push

* Post-processor failed: Bad exit status: 1

[smaddock]

Just running into this myself, although I don't think it's specific to ECR. Rather it seems to be that docker-push does not respect the Docker context set nor have a way to specify the Docker context, and only uses the default context. Colima (and other container runtimes) specify alternate contexts as can be seen by running docker context ls. Example:

❯ docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT
colima *   colima                                    unix:///Users/smaddock/.colima/default/docker.sock
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

This is why docker push works manually but the post processor does not. Note that this is a different issue than running Packer on a remote host... the host is still local; the socket is just in an alternate location.

[smaddock]

Documentation: https://docs.docker.com/engine/context/working-with-contexts/

If we could set the DOCKER_CONTEXT environment variable, or the --context command-line argument like we can with run_command on the builder, that would work.