Build docker image in parallel and credentials are removed before all images are push to ECR

hc-github-team-packer commented 2 years ago

This issue was originally opened by @kenit in https://github.com/hashicorp/packer/issues/11358 and has been migrated to this repository. The original issue description is below.

Overview of the Issue

I am trying to build two images in parallel base on two different version PHP image and to push artifacts into AWS ECR. When one of two artifacts is pushed to ECR, Packer will remove the login credential of ECR and break the push process of another artifact.

Packer version

1.7.7

Simplified Packer Template

source "docker" "php" {
  image  = "php:${var.php_version}-fpm"
  commit = true
}

source "docker" "php-cli" {
  image  = "php:${var.php_version}-cli"
  commit = true
}

build {
  sources = [
    "source.docker.php",
    "source.docker.php-cli"
  ]

  provisioner "shell" {
    inline = [
      "apt-get update",
      "apt-get install -y python3 git"
    ]
  }

  post-processors {
    post-processor "docker-tag" {
      repository = var.repository
      tags       = ["php-fpm"]
      only       = ["docker.php"]
    }

   post-processor "docker-tag" {
     repository = var.repository
     tags       = ["php-cli"]
     only       = ["docker.php-cli"]
   }

    post-processor "docker-push" {
      ecr_login    = true
      login_server = split("/", var.repository)[0]
    }
  }

}

Operating system and Environment details

OS: Ubuntu

Log Fragments and crash.log files

==> docker.php: Running post-processor:  (type docker-push)
    docker.php (docker-push): Fetching ECR credentials...
    docker.php (docker-push): Logging in...
    docker.php (docker-push): WARNING! Your password will be stored unencrypted in /var/lib/jenkins/.docker/config.json.
    docker.php (docker-push): Configure a credential helper to remove this warning. See
    docker.php (docker-push): https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    docker.php (docker-push): Login Succeeded
    docker.php (docker-push): Pushing: my.registry/test:php-fpm
    docker.php (docker-push): The push refers to repository [my.registry/test]
    docker.php (docker-push): 35a62a98fd04: Preparing
    docker.php (docker-push): a4a6079a9397: Preparing
    docker.php (docker-push): 08063f6b895d: Preparing
    docker.php (docker-push): 9de2214566a7: Preparing
    docker.php (docker-push): 09e59befc5ab: Preparing
    docker.php (docker-push): 18a6936c66e8: Preparing
    docker.php (docker-push): f8514c5993f9: Preparing
    docker.php (docker-push): 42cecd8ee840: Preparing
    docker.php (docker-push): d23471ea5612: Preparing
    docker.php (docker-push): 0392b5fdaffc: Preparing
    docker.php (docker-push): 4d3bf3167875: Preparing
    docker.php (docker-push): 42cecd8ee840: Waiting
    docker.php (docker-push): d23471ea5612: Waiting
    docker.php (docker-push): 0392b5fdaffc: Waiting
    docker.php (docker-push): 4d3bf3167875: Waiting
    docker.php (docker-push): 18a6936c66e8: Waiting
    docker.php (docker-push): f8514c5993f9: Waiting
    docker.php (docker-push): a4a6079a9397: Layer already exists
    docker.php (docker-push): 08063f6b895d: Layer already exists
    docker.php (docker-push): 9de2214566a7: Layer already exists
    docker.php (docker-push): 09e59befc5ab: Layer already exists
    docker.php (docker-push): 18a6936c66e8: Layer already exists
    docker.php (docker-push): d23471ea5612: Layer already exists
    docker.php (docker-push): 42cecd8ee840: Layer already exists
    docker.php (docker-push): f8514c5993f9: Layer already exists
    docker.php (docker-push): 0392b5fdaffc: Layer already exists
    docker.php (docker-push): 4d3bf3167875: Layer already exists
    docker.php-cli: changed: [default]
    docker.php-cli:
    docker.php-cli: TASK [Remove php source code] **************************************************
    docker.php-cli: changed: [default]
    docker.php-cli:
    docker.php-cli: PLAY RECAP *********************************************************************
    docker.php-cli: default                    : ok=12   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
    docker.php-cli:
==> docker.php-cli: Committing the container
    docker.php-cli: Image ID: sha256:7372332cddca1e524c0e52f29e5a618b287e9e7aea30af1536c17de3c1deecb7
==> docker.php-cli: Killing the container: 9bafc10fe1a2cc27bcda7e36ffc28b7d2954e09fa579258a86dc45933d7b2f45
==> docker.php-cli: Running post-processor:  (type docker-tag)
    docker.php-cli (docker-tag): Tagging image: sha256:7372332cddca1e524c0e52f29e5a618b287e9e7aea30af1536c17de3c1deecb7
    docker.php-cli (docker-tag): Repository: my.registry/test:php-cli
==> docker.php-cli: Running post-processor:  (type docker-push)
    docker.php-cli (docker-push): Fetching ECR credentials...
    docker.php-cli (docker-push): Logging in...
    docker.php-cli (docker-push): WARNING! Your password will be stored unencrypted in /var/lib/jenkins/.docker/config.json.
    docker.php-cli (docker-push): Configure a credential helper to remove this warning. See
    docker.php-cli (docker-push): https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    docker.php-cli (docker-push): Login Succeeded
    docker.php-cli (docker-push): Pushing: my.registry/test:php-cli
    docker.php-cli (docker-push): The push refers to repository [my.registry/test]
    docker.php-cli (docker-push): 2bbb3d367de9: Preparing
    docker.php-cli (docker-push): bf16543ffe11: Preparing
    docker.php-cli (docker-push): b4b4b803baee: Preparing
    docker.php-cli (docker-push): 3a9949a759cd: Preparing
    docker.php-cli (docker-push): b601465815f8: Preparing
    docker.php-cli (docker-push): 3aa87050994e: Preparing
    docker.php-cli (docker-push): 42cecd8ee840: Preparing
    docker.php-cli (docker-push): d23471ea5612: Preparing
    docker.php-cli (docker-push): 0392b5fdaffc: Preparing
    docker.php-cli (docker-push): 4d3bf3167875: Preparing
    docker.php-cli (docker-push): b601465815f8: Waiting
    docker.php-cli (docker-push): 3aa87050994e: Waiting
    docker.php-cli (docker-push): 42cecd8ee840: Waiting
    docker.php-cli (docker-push): d23471ea5612: Waiting
    docker.php-cli (docker-push): 0392b5fdaffc: Waiting
    docker.php-cli (docker-push): 4d3bf3167875: Waiting
    docker.php-cli (docker-push): 3a9949a759cd: Layer already exists
    docker.php-cli (docker-push): b4b4b803baee: Layer already exists
    docker.php-cli (docker-push): bf16543ffe11: Layer already exists
    docker.php-cli (docker-push): b601465815f8: Layer already exists
    docker.php-cli (docker-push): 42cecd8ee840: Layer already exists
    docker.php-cli (docker-push): 3aa87050994e: Layer already exists
    docker.php-cli (docker-push): d23471ea5612: Layer already exists
    docker.php-cli (docker-push): 0392b5fdaffc: Layer already exists
    docker.php-cli (docker-push): 4d3bf3167875: Layer already exists
    docker.php (docker-push): 35a62a98fd04: Pushed
    docker.php (docker-push): php-fpm: digest: sha256:2d39cd4bd30195d527eef4189bf11a6c38dd246d67df9b087dd4ab2762f78a63 size: 2622
    docker.php (docker-push): Pushing: my.registry/test:php-fpm
    docker.php (docker-push): The push refers to repository [my.registry/test]
    docker.php (docker-push): 35a62a98fd04: Preparing
    docker.php (docker-push): a4a6079a9397: Preparing
    docker.php (docker-push): 08063f6b895d: Preparing
    docker.php (docker-push): 9de2214566a7: Preparing
    docker.php (docker-push): 09e59befc5ab: Preparing
    docker.php (docker-push): 18a6936c66e8: Preparing
    docker.php (docker-push): f8514c5993f9: Preparing
    docker.php (docker-push): 42cecd8ee840: Preparing
    docker.php (docker-push): d23471ea5612: Preparing
    docker.php (docker-push): 0392b5fdaffc: Preparing
    docker.php (docker-push): 4d3bf3167875: Preparing
    docker.php (docker-push): 09e59befc5ab: Waiting
    docker.php (docker-push): 18a6936c66e8: Waiting
    docker.php (docker-push): f8514c5993f9: Waiting
    docker.php (docker-push): 42cecd8ee840: Waiting
    docker.php (docker-push): d23471ea5612: Waiting
    docker.php (docker-push): 0392b5fdaffc: Waiting
    docker.php (docker-push): 4d3bf3167875: Waiting
    docker.php (docker-push): 08063f6b895d: Layer already exists
    docker.php (docker-push): 9de2214566a7: Layer already exists
    docker.php (docker-push): a4a6079a9397: Layer already exists
    docker.php (docker-push): 35a62a98fd04: Layer already exists
    docker.php (docker-push): 09e59befc5ab: Layer already exists
    docker.php (docker-push): 18a6936c66e8: Layer already exists
    docker.php (docker-push): 42cecd8ee840: Layer already exists
    docker.php (docker-push): f8514c5993f9: Layer already exists
    docker.php (docker-push): d23471ea5612: Layer already exists
    docker.php (docker-push): 0392b5fdaffc: Layer already exists
    docker.php (docker-push): 4d3bf3167875: Layer already exists
    docker.php (docker-push): php-fpm: digest: sha256:2d39cd4bd30195d527eef4189bf11a6c38dd246d67df9b087dd4ab2762f78a63 size: 2622
    docker.php (docker-push): Logging out...
    docker.php (docker-push): Removing login credentials for my.registry
Build 'docker.php' finished after 3 minutes 24 seconds.
    docker.php-cli (docker-push): 2bbb3d367de9: Pushed
    docker.php-cli (docker-push): php-cli: digest: sha256:d261fc7ad33d6417ddc4ce4584dc7506c19a52dc908349f080f246217104f222 size: 2413
    docker.php-cli (docker-push): Pushing: my.registry/test:php-cli
    docker.php-cli (docker-push): The push refers to repository [my.registry/test]
    docker.php-cli (docker-push): 2bbb3d367de9: Preparing
    docker.php-cli (docker-push): bf16543ffe11: Preparing
    docker.php-cli (docker-push): b4b4b803baee: Preparing
    docker.php-cli (docker-push): 3a9949a759cd: Preparing
    docker.php-cli (docker-push): b601465815f8: Preparing
    docker.php-cli (docker-push): 3aa87050994e: Preparing
    docker.php-cli (docker-push): 42cecd8ee840: Preparing
    docker.php-cli (docker-push): d23471ea5612: Preparing
    docker.php-cli (docker-push): 0392b5fdaffc: Preparing
    docker.php-cli (docker-push): 4d3bf3167875: Preparing
    docker.php-cli (docker-push): no basic auth credentials
    docker.php-cli (docker-push): Logging out...
    docker.php-cli (docker-push): Removing login credentials for my.registry
Build 'docker.php-cli' errored after 3 minutes 33 seconds: 1 error(s) occurred:

* Post-processor failed: Bad exit status: 1

==> Wait completed after 3 minutes 33 seconds

==> Some builds didn't complete successfully and had errors:
--> docker.php-cli: 1 error(s) occurred:

* Post-processor failed: Bad exit status: 1

==> Builds finished. The artifacts of successful builds are:
--> docker.php: Imported Docker image: sha256:97d8a3f4708ccde2c9dce2a7988f3dd6df5462dc1deb42c464a67f23eca90854
--> docker.php: Imported Docker image: my.registry/test:php-fpm with tags my.registry/test:php-fpm

hertzsprung commented 2 years ago

-parallel-builds=1 works around this problem for me, so that each docker-push runs sequentially, logging in to ECR then logging out before the next docker-push.

nywilken commented 2 years ago

Hi @kenit @hertzsprung thanks again for reporting, this should be fixed by #96. The changes in the PR modify the docker-push post processor to use a temporary, isolated Docker client configuration directory per post-processor run, which should prevent Packer from creating shared credential files. There are some test binaries available via the link below if you would like to test the fix before the next release. https://app.circleci.com/pipelines/github/hashicorp/packer-plugin-docker/107/workflows/0599fc6d-d791-4430-980e-b23e86604963/jobs/1433/artifacts

hashicorp / packer-plugin-docker