Packer freezes in the terminal window until terminal window is resized

thomastthai commented 11 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

When building with packer, the terminal window would stop updating at random places with additional lines of text until the terminal window is resized to different dimensions. For example:

==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Image type: raspberrypi
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Retrieving Image
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Trying images/rpi-k3s-base-raspios-bookworm-armhf.img.gz
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Trying images/rpi-k3s-base-raspios-bookworm-armhf.img.gz?checksum=sha256%3A893a637a3d447974255a25242818ca66e5b37b31029488017f184cdb13a5e59c
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: images/rpi-k3s-base-raspios-bookworm-armhf.img.gz?checksum=sha256%3A893a637a3d447974255a25242818ca66e5b37b31029488017f184cdb13a5e59c => /vagrant/images/rpi-k3s-base-raspios-bookworm-armhf.img.gz
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Copying source image.
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Image is a gzip file.
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   29.41 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   29.67 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   27.71 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   22.98 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   25.37 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   19.76 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   23.83 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   22.53 MB/s
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Speed:   23.37 MB/s
... freeze here ...

After the "freeze" and the terminal window is resized, it updates the next lines right away and continues with the build. If the resize doesn't happen, it could result in an unexpected EOF error similar to:

Build 'k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf' errored after 3 minutes 41 seconds: unexpected EOF

or

Build 'k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf' errored after 4 minutes 50 seconds: 2 error(s) occurred:

* Post-processor failed: unexpected EOF
* Error destroying builder artifact: reading body msgpack decode error [pos 973]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

Another build that froze but didn't have an unexpected EOF error. I didn't resize the terminal window and let it sit until it timed out:

==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
    k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: k3s installation finished.
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: fuser -k /tmp/armimg-3512686872
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: Running post-processor:  (type compress)
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64 (compress): Using pgzip compression with 2 cores for images/rpi-k3s-agent-raspios-bookworm-arm64.img.gz
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64 (compress): Archiving images/rpi-k3s-agent-raspios-bookworm-arm64.img with pgzip
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64 (compress): Archive images/rpi-k3s-agent-raspios-bookworm-arm64.img.gz completed
==> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: Running post-processor:  (type checksum)
Build 'k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64' errored after 25 minutes 4 seconds: 1 error(s) occurred:

* Error destroying builder artifact: connection is shut down; bad artifact: []string(nil)

==> Wait completed after 51 minutes 49 seconds

==> Some builds didn't complete successfully and had errors:
--> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-arm64: 1 error(s) occurred:

* Error destroying builder artifact: connection is shut down; bad artifact: []string(nil)

==> Builds finished. The artifacts of successful builds are:
--> k3s-server.arm-image.rpi-k3s-server-raspios-bookworm-armhf: compressed artifacts in: images/rpi-k3s-server-raspios-bookworm-armhf.img.gz
--> k3s-server.arm-image.rpi-k3s-server-raspios-bookworm-armhf: Created artifact from files: images/rpi-k3s-server-raspios-bookworm-armhf.img.gz
--> k3s-server.arm-image.rpi-k3s-server-raspios-bookworm-arm64: compressed artifacts in: images/rpi-k3s-server-raspios-bookworm-arm64.img.gz
--> k3s-server.arm-image.rpi-k3s-server-raspios-bookworm-arm64: Created artifact from files: images/rpi-k3s-server-raspios-bookworm-arm64.img.gz
--> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: compressed artifacts in: images/rpi-k3s-agent-raspios-bookworm-armhf.img.gz
--> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: Created artifact from files: images/rpi-k3s-agent-raspios-bookworm-armhf.img.gz
vagrant@debian12:/vagrant$

Reproduction Steps

vagrant@debian12:/vagrant$ export PACKER_LOG=1 && export PACKER_LOG_PATH="/vagrant/packer.log"
vagrant@debian12:/vagrant$ sudo packer build -debug -parallel-builds=1 -on-error=ask ./packer/server-agent

Packer version

1.9.4

Simplified Packer Template

build.pkr.hcl

build {
  name = "k3s-server"

  source "source.arm-image.raspios_bookworm_armhf" {
    name = "rpi-k3s-server-raspios-bookworm-armhf"
    output_filename = "images/${source.name}.img"
  }

  source "source.arm-image.raspios_bookworm_arm64" {
    name = "rpi-k3s-server-raspios-bookworm-arm64"
    output_filename = "images/${source.name}.img"

  }

  provisioner "shell" {
    scripts = [
      "scripts/install-k3s-server.sh"
    ]
  }

  post-processors {
    post-processor "compress" {
      output = "images/${source.name}.img.gz"
    }
    post-processor "checksum" {
      checksum_types = ["sha256"]
      output = "images/${source.name}.img.gz.sha256"
    }
  }
}

build {
  name = "k3s-agent"

  source "source.arm-image.raspios_bookworm_armhf" {
    name = "rpi-k3s-agent-raspios-bookworm-armhf"
    output_filename = "images/${source.name}.img"
  }

  source "source.arm-image.raspios_bookworm_arm64" {
    name = "rpi-k3s-agent-raspios-bookworm-arm64"
    output_filename = "images/${source.name}.img"
  }

  provisioner "shell" {
    scripts = [
      "scripts/install-k3s-agent.sh"
    ]
  }

  post-processors {
    post-processor "compress" {
      output = "images/${source.name}.img.gz"
    }
    post-processor "checksum" {
      checksum_types = ["sha256"]
      output = "images/${source.name}.img.gz.sha256"
    }
  }

}

sources.pkr.hcl

source "arm-image" "raspios_bookworm_armhf" {

  image_type      = "raspberrypi"
  iso_url         = "images/rpi-k3s-base-raspios-bookworm-armhf.img.gz"
  iso_checksum    = "file:images/rpi-k3s-base-raspios-bookworm-armhf.img.gz.sha256"
}

source "arm-image" "raspios_bookworm_arm64" {
  image_type      = "raspberrypi"
  iso_url         = "images/rpi-k3s-base-raspios-bookworm-arm64.img.gz"
  iso_checksum    = "file:images/rpi-k3s-base-raspios-bookworm-arm64.img.gz.sha256"
  qemu_binary     = "qemu-aarch64-static"
}

plugins.pkr.hcl

packer {
  required_plugins {
    arm-image = {
      version = ">= 0.2.5"
      source  = "github.com/solo-io/arm-image"
    }
  }
}

Operating system and Environment details

Host machine: Windows 11 AMD64 Host machine terminal: Mintty 3.6.5 (xterm-256color emulation) via Git Bash 2.27.0 Guest VM: via Vagrant 2.4.0, VirtualBox 7.0, Debian 12

Log Fragments and crash.log files

The errors will vary depending on where the "freeze" happened, but always include that unexpected EOF error. Below is another example of an error from one of the builds.

==> Some builds didn't complete successfully and had errors:
--> k3s-agent.arm-image.rpi-k3s-agent-raspios-bookworm-armhf: 2 error(s) occurred. 

* Post-processor failed: unexpected EOF
* Error destroying builder artifact: reading body msgpack decode error [pos 973]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

nywilken commented 11 months ago

Hi @thomastthai the terminal freezing seems unrelated to Packer as it is most likely an issue with the terminal and Host operating causing the terminal to lock up. I recommend looking at the load on your machine or the Windows forums for open issues with terminal resizing.

That said, the error that is being reported unexpected EOF seems to me that the post-processor causing the failure is either being killed by the build host - we've seen this with OOM errors. Or that the post-processor has a bug.

Have you tried running the build without the post-processor block to see if the build executes successfully?

To help identify what might be going on I would remove the post-processors run a build. If that succeeds I would then add back one post-processor to see with one fails and use the debug logs PACKER_LOG=1 packer build source.pkr.hcl to see what is happening when the post-processor fails.

nywilken commented 11 months ago

Hi there :wave:, I'm going to close this issue since there hasn't been any updates to it. But if you are still running into issues please feel free to leave a comment on the issue and we will gladly reopen.

thomastthai commented 11 months ago

I am still testing and will report.

On Fri, Nov 17, 2023, 3:31 AM Wilken Rivera @.***> wrote:

Hi there 👋, I'm going to close this issue since there hasn't been any updates to it. But if you are still running into issues please feel free to leave a comment on the issue and we will gladly reopen.

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/packer/issues/12685#issuecomment-1816210955, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD6EIZOL7NNY7WYMJ2RMHTTYE5DJ7AVCNFSM6AAAAAA7B6ORGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJWGIYTAOJVGU . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 10 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / packer