search

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.06k stars 3.32k forks source link

Packer does not shutdown cleanly on SIGTERM #5256

Closed mdub closed 6 years ago

mdub commented 7 years ago

I'm using Packer to build an AMI.

I sent Packer a SIGTERM (e.g. using kill $packer_pid), expecting it to clean up, in the same way it does in response to Ctrl-C, i.e.

Unfortunately, it not cleanup, but instead appeared re-try parts of the build process.

Details

Template

{
  "description": "Build Mike's test AMI",
  "variables": {
    "ami_name": "{{env `AMI_NAME`}}",
    "base_ami": "{{env `BASE_AMI`}}"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "region": "ap-southeast-2",
      "source_ami": "{{user `base_ami`}}",
      "instance_type": "c3.large",
      "ssh_username": "ubuntu",
      "ssh_timeout": "5m",
      "ssh_port": "22",
      "ami_name": "{{user `ami_name`}}",
      "run_tags": {
        "target_ami_name": "{{user `ami_name`}}"
      },
      "subnet_id": "subnet-7917fe1c",
      "vpc_id": "vpc-399e7a5c"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [ "sudo apt-get -yq dist-upgrade" ]
    },
    {
      "type": "shell",
      "inline": [ "sudo reboot" ]
    }
  ]
}

Command output

amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name...
    amazon-ebs: Found Image ID: ami-e94e5e8a
==> amazon-ebs: Creating temporary keypair: packer_5993cf3e-16ac-68ed-df19-3b9d4e11a80f
==> amazon-ebs: Creating temporary security group for this instance: packer_5993cf40-5236-ab92-5d6f-bf8466f45580
==> amazon-ebs: Authorizing access to port 22 on the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
    amazon-ebs: Instance ID: i-0bf866829e209fc03
==> amazon-ebs: Waiting for instance (i-0bf866829e209fc03) to become ready...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "target_ami_name": "mwilliams-debug--201708161451"
    amazon-ebs: Adding tag: "Name": "Packer Builder"
==> amazon-ebs: Waiting for SSH to become available...

<--- "kill $packer_pid" happened here ---

2017/08/16 14:51:40 [INFO] Packer version: 1.0.4
2017/08/16 14:51:40 Packer Target OS/Arch: darwin amd64
2017/08/16 14:51:40 Built with Go Version: go1.8.3
2017/08/16 14:51:40 Detected home directory from env var: /Users/mike_williams
2017/08/16 14:51:40 Using internal plugin for azure-arm
2017/08/16 14:51:40 Using internal plugin for hyperv-iso
2017/08/16 14:51:40 Using internal plugin for parallels-iso
2017/08/16 14:51:40 Using internal plugin for triton
2017/08/16 14:51:40 Using internal plugin for alicloud-ecs
2017/08/16 14:51:40 Using internal plugin for amazon-chroot
2017/08/16 14:51:40 Using internal plugin for cloudstack
2017/08/16 14:51:40 Using internal plugin for file
2017/08/16 14:51:40 Using internal plugin for openstack
2017/08/16 14:51:40 Using internal plugin for vmware-iso
2017/08/16 14:51:40 Using internal plugin for qemu
2017/08/16 14:51:40 Using internal plugin for amazon-ebs
2017/08/16 14:51:40 Using internal plugin for amazon-ebssurrogate
2017/08/16 14:51:40 Using internal plugin for amazon-ebsvolume
2017/08/16 14:51:40 Using internal plugin for docker
2017/08/16 14:51:40 Using internal plugin for null
2017/08/16 14:51:40 Using internal plugin for parallels-pvm
2017/08/16 14:51:40 Using internal plugin for profitbricks
2017/08/16 14:51:40 Using internal plugin for virtualbox-ovf
2017/08/16 14:51:40 Using internal plugin for vmware-vmx
2017/08/16 14:51:40 Using internal plugin for amazon-instance
2017/08/16 14:51:40 Using internal plugin for digitalocean
2017/08/16 14:51:40 Using internal plugin for googlecompute
2017/08/16 14:51:40 Using internal plugin for oneandone
2017/08/16 14:51:40 Using internal plugin for virtualbox-iso
2017/08/16 14:51:40 Using internal plugin for salt-masterless
2017/08/16 14:51:40 Using internal plugin for ansible
2017/08/16 14:51:40 Using internal plugin for chef-client
2017/08/16 14:51:40 Using internal plugin for chef-solo
2017/08/16 14:51:40 Using internal plugin for file
2017/08/16 14:51:40 Using internal plugin for shell
2017/08/16 14:51:40 Using internal plugin for ansible-local
2017/08/16 14:51:40 Using internal plugin for converge
2017/08/16 14:51:40 Using internal plugin for powershell
2017/08/16 14:51:40 Using internal plugin for puppet-masterless
2017/08/16 14:51:40 Using internal plugin for shell-local
2017/08/16 14:51:40 Using internal plugin for windows-restart
2017/08/16 14:51:40 Using internal plugin for puppet-server
2017/08/16 14:51:40 Using internal plugin for windows-shell
2017/08/16 14:51:40 Using internal plugin for alicloud-import
2017/08/16 14:51:40 Using internal plugin for artifice
2017/08/16 14:51:40 Using internal plugin for docker-push
2017/08/16 14:51:40 Using internal plugin for manifest
2017/08/16 14:51:40 Using internal plugin for vagrant
2017/08/16 14:51:40 Using internal plugin for vagrant-cloud
2017/08/16 14:51:40 Using internal plugin for amazon-import
2017/08/16 14:51:40 Using internal plugin for docker-tag
2017/08/16 14:51:40 Using internal plugin for shell-local
2017/08/16 14:51:40 Using internal plugin for googlecompute-export
2017/08/16 14:51:40 Using internal plugin for atlas
2017/08/16 14:51:40 Using internal plugin for checksum
2017/08/16 14:51:40 Using internal plugin for compress
2017/08/16 14:51:40 Using internal plugin for docker-import
2017/08/16 14:51:40 Using internal plugin for docker-save
2017/08/16 14:51:40 Using internal plugin for vsphere
2017/08/16 14:51:40 Detected home directory from env var: /Users/mike_williams
2017/08/16 14:51:40 Attempting to open config file: /Users/mike_williams/.packerconfig
2017/08/16 14:51:40 [WARN] Config file doesn't exist: /Users/mike_williams/.packerconfig
2017/08/16 14:51:40 Packer config: &{DisableCheckpoint:false DisableCheckpointSignature:false PluginMinPort:10000 PluginMaxPort:25000 Builders:map[docker:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-docker profitbricks:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-profitbricks amazon-instance:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-instance parallels-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-iso triton:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-triton cloudstack:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-cloudstack qemu:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-qemu amazon-ebssurrogate:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebssurrogate azure-arm:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-azure-arm amazon-ebs:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebs null:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-null virtualbox-ovf:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-ovf googlecompute:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-googlecompute oneandone:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oneandone alicloud-ecs:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-alicloud-ecs file:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-file openstack:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-openstack amazon-ebsvolume:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebsvolume parallels-pvm:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-pvm virtualbox-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso hyperv-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-hyperv-iso amazon-chroot:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-chroot vmware-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-iso vmware-vmx:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-vmx digitalocean:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-digitalocean] PostProcessors:map[docker-save:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-save docker-push:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-push docker-tag:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-tag shell-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-shell-local vsphere:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vsphere alicloud-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-alicloud-import googlecompute-export:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-googlecompute-export atlas:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-atlas compress:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-compress docker-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-import vagrant-cloud:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant-cloud amazon-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-amazon-import checksum:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-checksum artifice:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-artifice manifest:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-manifest vagrant:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant] Provisioners:map[powershell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-powershell puppet-masterless:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-masterless salt-masterless:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-salt-masterless ansible:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible chef-client:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-client converge:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-converge shell-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell-local windows-shell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-shell shell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell windows-restart:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-restart puppet-server:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-server chef-solo:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-solo file:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-file ansible-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible-local]}
2017/08/16 14:51:40 Detected home directory from env var: /Users/mike_williams
2017/08/16 14:51:40 Setting cache directory: /Users/mike_williams/Code/git.realestate.com.au/mwilliams/packer-debug/packer_cache
2017/08/16 14:51:40 Detected home directory from env var: /Users/mike_williams
2017/08/16 14:51:40 Loading builder: amazon-ebs
2017/08/16 14:51:40 Plugin could not be found. Checking same directory as executable.
2017/08/16 14:51:40 Current exe path: /usr/local/bin/packer
2017/08/16 14:51:40 Creating plugin client for path: /usr/local/bin/packer
2017/08/16 14:51:40 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-builder-amazon-ebs"}
2017/08/16 14:51:40 Waiting for RPC address for: /usr/local/bin/packer
2017/08/16 14:51:41 Loading provisioner: shell
2017/08/16 14:51:41 Plugin could not be found. Checking same directory as executable.
2017/08/16 14:51:41 Current exe path: /usr/local/bin/packer
2017/08/16 14:51:41 Creating plugin client for path: /usr/local/bin/packer
2017/08/16 14:51:41 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell"}
2017/08/16 14:51:41 Waiting for RPC address for: /usr/local/bin/packer
2017/08/16 14:51:41 Loading provisioner: shell
2017/08/16 14:51:41 Plugin could not be found. Checking same directory as executable.
2017/08/16 14:51:41 Current exe path: /usr/local/bin/packer
2017/08/16 14:51:41 Creating plugin client for path: /usr/local/bin/packer
2017/08/16 14:51:41 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell"}
2017/08/16 14:51:41 Waiting for RPC address for: /usr/local/bin/packer
2017/08/16 14:51:41 ui: amazon-ebs output will be in this color.
amazon-ebs output will be in this color.
2017/08/16 14:51:41 ui: 

2017/08/16 14:51:41 Build debug mode: false
2017/08/16 14:51:41 Force build: false
2017/08/16 14:51:41 On error: 
2017/08/16 14:51:41 Preparing build: amazon-ebs
2017/08/16 14:51:41 Waiting on builds to complete...
2017/08/16 14:51:41 Starting build run: amazon-ebs
2017/08/16 14:51:41 Running builder: amazon-ebs
2017/08/16 14:51:41 [INFO] (telemetry) Starting builder amazon-ebs
2017/08/16 14:51:41 ui: ==> amazon-ebs: Prevalidating AMI Name...
==> amazon-ebs: Prevalidating AMI Name...
2017/08/16 14:51:42 ui:     amazon-ebs: Found Image ID: ami-e94e5e8a
    amazon-ebs: Found Image ID: ami-e94e5e8a
2017/08/16 14:51:42 ui: ==> amazon-ebs: Creating temporary keypair: packer_5993cf5d-0661-2fa7-0c25-cd515f7b666a
==> amazon-ebs: Creating temporary keypair: packer_5993cf5d-0661-2fa7-0c25-cd515f7b666a
2017/08/16 14:51:42 ui: ==> amazon-ebs: Creating temporary security group for this instance: packer_5993cf5e-6ee4-2e85-66c4-462e5e28df1a
==> amazon-ebs: Creating temporary security group for this instance: packer_5993cf5e-6ee4-2e85-66c4-462e5e28df1a
2017/08/16 14:51:42 ui: ==> amazon-ebs: Authorizing access to port 22 on the temporary security group...
==> amazon-ebs: Authorizing access to port 22 on the temporary security group...
2017/08/16 14:51:42 ui: ==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Launching a source AWS instance...
2017/08/16 14:51:43 ui:     amazon-ebs: Instance ID: i-04d33a84177cdc4e0
    amazon-ebs: Instance ID: i-04d33a84177cdc4e0
2017/08/16 14:51:43 ui: ==> amazon-ebs: Waiting for instance (i-04d33a84177cdc4e0) to become ready...
==> amazon-ebs: Waiting for instance (i-04d33a84177cdc4e0) to become ready...
2017/08/16 14:51:54 ui: ==> amazon-ebs: Adding tags to source instance
==> amazon-ebs: Adding tags to source instance
2017/08/16 14:51:54 ui:     amazon-ebs: Adding tag: "target_ami_name": "mwilliams-debug--201708161451"
    amazon-ebs: Adding tag: "target_ami_name": "mwilliams-debug--201708161451"
2017/08/16 14:51:54 ui:     amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Adding tag: "Name": "Packer Builder"
2017/08/16 14:51:54 ui: ==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Waiting for SSH to become available...
2017/08/16 14:56:54 ui error: ==> amazon-ebs: Timeout waiting for SSH.
==> amazon-ebs: Timeout waiting for SSH.
2017/08/16 14:56:54 ui: ==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Terminating the source AWS instance...
2017/08/16 14:57:19 ui: ==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: Cleaning up any extra volumes...
2017/08/16 14:57:19 ui: ==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: No volumes to clean up, skipping
2017/08/16 14:57:19 ui: ==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary security group...
2017/08/16 14:57:19 ui: ==> amazon-ebs: Deleting temporary keypair...
==> amazon-ebs: Deleting temporary keypair...
2017/08/16 14:57:19 [INFO] (telemetry) ending amazon-ebs
2017/08/16 14:57:19 [INFO] (telemetry) found error: Timeout waiting for SSH.
2017/08/16 14:57:19 ui error: Build 'amazon-ebs' errored: Timeout waiting for SSH.
Build 'amazon-ebs' errored: Timeout waiting for SSH.
2017/08/16 14:57:19 Builds completed. Waiting on interrupt barrier...
2017/08/16 14:57:19 machine readable: error-count []string{"1"}
2017/08/16 14:57:19 ui error: 
==> Some builds didn't complete successfully and had errors:

==> Some builds didn't complete successfully and had errors:
2017/08/16 14:57:19 machine readable: amazon-ebs,error []string{"Timeout waiting for SSH."}
2017/08/16 14:57:19 ui error: --> amazon-ebs: Timeout waiting for SSH.
--> amazon-ebs: Timeout waiting for SSH.
2017/08/16 14:57:19 ui: 
==> Builds finished but no artifacts were created.

==> Builds finished but no artifacts were created.
2017/08/16 14:57:20 [WARN] (telemetry) Error finalizing report. This is safe to ignore. Post https://checkpoint-api.hashicorp.com/v1/telemetry/packer: context deadline exceeded
2017/08/16 14:57:20 waiting for all plugin processes to complete...

Final state

I can see that some packer process are still running

$ ps -f | grep packer | grep -v grep
1491561753 97470 97199   0  2:51pm ttys001    0:00.81 packer build --var ami_name=mwilliams-debug--201708161451 --var base_ami=ami-e94e5e8a packer-template.json
1491561753 97523     1   0  2:51pm ttys001    0:00.30 /usr/local/bin/packer plugin packer-builder-amazon-ebs
1491561753 97531     1   0  2:51pm ttys001    0:00.03 /usr/local/bin/packer plugin packer-provisioner-shell
1491561753 97539     1   0  2:51pm ttys001    0:00.03 /usr/local/bin/packer plugin packer-provisioner-shell
mdub commented 7 years ago

I see an attempt to fix this, in https://github.com/hashicorp/packer/commit/50fef50e4be908929daf5b917e95e48119678792.

Unfortunately, it doesn't appear to be working as planned. My hunch is that SIGTERM needs to be treated differently from SIGINT. I think that on Ctrl-C, a shell will send interrupt to all processes in the process group, so packer need not explicitly propagate SIGINT to child processes. But SIGTERM (via kill) is typically only sent to the master process.

trodemaster commented 7 years ago

In my automation around packer this works for stopping packer with kill and behaving like Crtl-C.

kill -s 15 `pgrep -xo packer`

It would be nice if kill $packer_pid just worked as expected.

mwhooker commented 7 years ago

to be clear, you're expecting sigterm to be handled the same way as sigint? I looked through the code and whatever sigterm handling there used to be got removed in a refactor ages ago. Is there a reason sigint isn't sufficient?

It seems to me that sigterm should make sure all child processes have exited successfully, but not wait for resources to be cleaned up

mwhooker commented 7 years ago

from the original issue

[handling sigterm] allows for better integration with various tools expecting standard Unix semantics (for instance Jenkins, where cancelling a job will cause SIGTERM to be sent to the process).

which would make sense for cleaning up resources

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.