hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.11k stars 3.33k forks source link

Packer inconsistent behavior with Ansible inside / outside container #11789

Closed sc250024 closed 2 years ago

sc250024 commented 2 years ago

Community Note

Overview of the Issue

Packer behaves inconsistently when Packer with an Ansible provisioner is called from inside of a Docker container. Basically, a very simple build of an Ubuntu 22.04 AMI with a very simple ansible provisioner fails consistently. However, when this same build is performed outside of a container (using tools installed on my local OS), it works fine.

This could be related to Ansible itself, but I am starting this issue here.

Reproduction Steps

Start by placing the following 5 files in a directory:

ansible.cfg ```config [defaults] ask_pass = False ask_sudo_pass = False display_skipped_hosts = True host_key_checking = False local_tmp = /tmp/.ansible remote_tmp = /tmp/.ansible remote_user = root roles_path = /etc/ansible/roles [privilege_escalation] become = True become_ask_pass = False become_method = sudo become_user = root [connection] control_path_dir = /tmp/.ansible/cp ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes usetty = True [diff] always = yes ```
Dockerfile ```dockerfile ######## # Base # ######## FROM docker.io/library/alpine:latest AS tools ARG PACKER_VERSION=1.8.0 RUN apk add --no-cache ca-certificates curl jq zip RUN curl -fsSL -O "https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip" \ && unzip "packer_${PACKER_VERSION}_linux_amd64.zip" -d /usr/bin \ && chown root:root /usr/bin/packer \ && chmod 0755 /usr/bin/packer ############ # Main App # ############ FROM docker.io/library/python:3.10-slim AS app # Change working dir ENV BUILD_DIR /build WORKDIR ${BUILD_DIR} # Install SSH RUN apt-get -qy update \ && apt-get -qy install --no-install-recommends openssh-client # Install Ansible RUN pip3 install --no-cache-dir ansible==5.7.1 # Copy tools from tools stage COPY --from=tools /usr/bin/packer /usr/bin/packer RUN packer version # Copy Ansible config RUN mkdir -p /etc/ansible COPY ansible.cfg /etc/ansible/ansible.cfg COPY . . # Initialize Packer plugins RUN packer init . ```
docker-compose.yaml ```yaml name: ami-builder services: base-ami: build: context: . dockerfile: Dockerfile command: - packer - build - -force - -on-error=cleanup - . environment: AWS_ACCESS_KEY_ID: AWS_DEFAULT_REGION: us-east-1 AWS_SECRET_ACCESS_KEY: PACKER_LOG: "10" stdin_open: true tty: true ```
build.pkr.hcl ```hcl packer { required_version = ">= 1.8.0" required_plugins { amazon = { version = ">= 1.1.0, < 2.0.0" source = "github.com/hashicorp/amazon" } } } data "amazon-ami" "ubuntu-jammy-amd64-latest" { filters = { name = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["099720109477"] region = "us-east-1" } source "amazon-ebs" "ubuntu-jammy-amd64-latest" { ami_name = "test-build" associate_public_ip_address = true communicator = "ssh" force_delete_snapshot = true force_deregister = true instance_type = "t3.large" region = "us-east-1" source_ami = data.amazon-ami.ubuntu-jammy-amd64-latest.id ssh_clear_authorized_keys = true ssh_username = "ubuntu" temporary_key_pair_type = "ed25519" } build { name = "test-build" sources = ["source.amazon-ebs.ubuntu-jammy-amd64-latest"] provisioner "ansible" { extra_arguments = ["-vv"] playbook_file = "${path.root}/main.yml" } } ```
main.yml ```yaml --- - become: true hosts: all tasks: - ansible.builtin.apt: update_cache: true name: update apt cache ```

You can see these 5 files are for Docker, a main.yml which is a very simple Ansible playbook (which just updates the APT cache), the Ansible config ansible.cfg, and the Packer build.pkr.hcl file which contains the basic instructions to get going. Modify them as needed, replacing things like the AWS environment variables to make sure you can authenticate.

Basically, if you run docker-compose up --build, you'll get the following error at the very end:

ami-builder-base-ami-1  |     test-build.amazon-ebs.ubuntu-jammy-amd64-latest: fatal: [default]: FAILED! => {"changed": false, "msg": "Failed to lock apt for exclusive operation: Failed to lock directory /var/lib/apt/lists/: E:Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)"}
ami-builder-base-ami-1  |     test-build.amazon-ebs.ubuntu-jammy-amd64-latest:
ami-builder-base-ami-1  |     test-build.amazon-ebs.ubuntu-jammy-amd64-latest: PLAY RECAP *********************************************************************
ami-builder-base-ami-1  |     test-build.amazon-ebs.ubuntu-jammy-amd64-latest: default                    : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0

However, running the exact same command locally (using same versions as in the container) you get the following:

2022/05/18 15:26:15 packer-provisioner-ansible plugin: [INFO] 0 bytes written for 'stdin'
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: changed: [default] => {"cache_update_time": 1652880372, "cache_updated": true, "changed": true}
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: META: ran handlers
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: META: ran handlers
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest:
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: PLAY RECAP *********************************************************************
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: default                    : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest:
2022/05/18 15:26:16 packer-provisioner-ansible plugin: shutting down the SSH proxy
2022/05/18 15:26:16 [INFO] (telemetry) ending ansible
==> test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Trying to remove ephemeral keys from authorized_keys files
==> test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Stopping the source instance...
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Stopping instance
==> test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Waiting for the instance to stop...
==> test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Creating AMI test-build from instance i-06f370cd52584527b
    test-build.amazon-ebs.ubuntu-jammy-amd64-latest: AMI: ami-045e1a31c3b77bc8d
==> test-build.amazon-ebs.ubuntu-jammy-amd64-latest: Waiting for AMI to become ready...

Versions

/usr/local/Cellar/ansible/5.7.1/libexec/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
ansible [core 2.12.5]
  config file = None
  configured module search path = ['/Users/my-username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/5.7.1/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/my-username/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, Apr 26 2022, 19:43:24) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.1.2
  libyaml = True
ansible [core 2.12.5]
  config file = /build/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, May 11 2022, 07:26:18) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True

Simplified Packer Template

build.pkr.hcl ```hcl packer { required_version = ">= 1.8.0" required_plugins { amazon = { version = ">= 1.1.0, < 2.0.0" source = "github.com/hashicorp/amazon" } } } data "amazon-ami" "ubuntu-jammy-amd64-latest" { filters = { name = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["099720109477"] region = "us-east-1" } source "amazon-ebs" "ubuntu-jammy-amd64-latest" { ami_name = "test-build" associate_public_ip_address = true communicator = "ssh" force_delete_snapshot = true force_deregister = true instance_type = "t3.large" region = "us-east-1" source_ami = data.amazon-ami.ubuntu-jammy-amd64-latest.id ssh_clear_authorized_keys = true ssh_username = "ubuntu" temporary_key_pair_type = "ed25519" } build { name = "test-build" sources = ["source.amazon-ebs.ubuntu-jammy-amd64-latest"] provisioner "ansible" { extra_arguments = ["-vv"] playbook_file = "${path.root}/main.yml" } } ```

Operating system and Environment details

Log Fragments and crash.log files

The following gist contains the successful build (from local OS), and unsuccessful build (inside container): https://gist.github.com/sc250024/2d1dd67fd66b676e6c0f129eb75c84d3

nywilken commented 2 years ago

Hi 👋 thanks for reaching out.

This doesn't seem like a Packer related issue, as it does work when you build outside of the container. For general questions we recommend reaching out to the community forum for greater visibility. As the GitHub issue tracker is only watched by a small subset of maintainers and is really reserved for bugs and enhancements, you'll have a better chance of finding someone who can help you in the forum.

sc250024 commented 2 years ago

If anyone finds this in the future, the reason that this was not working is because it was running as root. In order to fix this, I simply added the following lines to the Dockerfile to make sure that it worked:

RUN useradd -ms /bin/bash packer
USER packer
github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.