failing to ssh into a FreeBSD AMI post packer build

[ghost]

This issue was originally opened by @david-peters-aitch2o as hashicorp/packer#10854. It was migrated here as a result of the Packer plugin split. The original body of the issue is below.

---

Overview of the Issue

with an extremely simple build, booting the ec2-instance it will fail to ssh in

Reproduction Steps

build with the following file:

{
  "variables": {
    "DB_USERNAME":        "{{env `DB_USERNAME`}}",
    "DB_PASSWORD":        "{{env `DB_PASSWORD`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "region": "ap-southeast-2",
    "profile": "test-workspace-dev",
    "source_ami": "ami-0e0a9bebd811801a4",
    "instance_type": "t2.medium",
    "ssh_username": "ec2-user",
    "ami_name": "instance {{timestamp}}",
    "ssh_timeout": "15m",
    "user_data_file": "user_data.txt"
    }]      
  }
}

user_data.txt file contains

#!/bin/sh
env ASSUME_ALWAYS_YES=YES pkg install sudo python bash bash-completion

deploy from above AMI and trying to ssh in with your keys will fail, deploy from the ami-0e0a9bebd811801a4 AMI, ssh with your keys and it works

Packer version

[INFO] Packer version: 1.7.0 [go1.15.8 linux amd64]

Simplified Packer Buildfile

see above

Operating system and Environment details

OS is FreeBSD 12 base AMI is ami-0e0a9bebd811801a4

Log Fragments and crash.log files

logs are fine as it builds as expected

[david-peters-aitch2o]

I have spent a little time on this to get to the bottom of it, using ansible provisioner I created a user and added it to wheel and enable wheel to sudo passwordless. I was then able to login with that user and look at the .ssh dir for ec2-user

root@freebsd:/home/ec2-user/.ssh # cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXvwmIWEQpL5JwnKex2OOWP1cwMt2BkZWM4oizK/cCMLDlbfX0Im1z4yLR1wztRobgnbLYMeL1KdmpCbX+DXw69xfIcjstbD58/ssmRyJmQDExVvhNOOpBCbpZzZqJGKTeY36WmUc2gGUwwNjxDO15+JrEx2lPc2Dzi8hiengHbiq+X0dBG4Kh2dVAVXpdbx8kpOF7ylFAlboiHpSeexDEbAUmo6agBD3c2SD1rQ8WMfqjX5eUqw8mMkck8xBD8vYjTZF565q6vJV1a6g11VD6UzfTUl9wIDHPK/BzNBRegNAx45zxj5oVGVlcYut8QPWpGt4cylmbEoJ+vMX5678R packer_6081191b-2d86-5b40-8f9e-4d2335f822b9 root@freebsd:/home/ec2-user/.ssh #

It seems Packer does not clean up after itself and the ec2-user remains with the ssh key from when it was built

[Probotect0r]

I am running into the same issue. I am building an AMI from the official FreeBSD 13 AMI, but not able to ssh into the built AMI even though ssh works with the source AMI.

My minimal packer script:

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "freebsd-13" {
  ami_name      = "turnserver-freebsd-${formatdate("YYYYMMDD-hhmm", timestamp())}"
  instance_type = "t2.micro"
  region        = "us-east-2"
  ssh_username  = "ec2-user"
  ssh_clear_authorized_keys = true
  source_ami    = "ami-023aa35d4157222d3"
}

build {
  sources = [
    "source.amazon-ebs.freebsd-13"
  ]

}

I tried using the ssh_clear_authorized_key option, but the build output reports the following error:

==> amazon-ebs.freebsd-13: Trying to remove ephemeral keys from authorized_keys files
==> amazon-ebs.freebsd-13: sh: sudo: not found
==> amazon-ebs.freebsd-13: sh: sudo: not found
==> amazon-ebs.freebsd-13: Stopping the source instance...

[Probotect0r]

I tried adding echo "" > ~/.ssh/authorized_keys to a shell script run using the shell provisioner, and still no luck. So I don't think the issue is that the temporary packer key is not being cleaned up. There is something else going on.

[david-peters-aitch2o]

It might be a permission issue, I actually never went back to check but I will, out of curiosity if nothing else.

clear the file and set permissions to I believe 400

edit: the packer key was there for me, I remember that part, see my first post

[Probotect0r]

Sorry, I meant that the key is definitely not being cleaned up, but I don't think that is preventing ssh into the instance post-build. Because I tried clearing the key as the last step of my provisioning script, and I am still not able to ssh into the instance when I launch it. I'll try setting the permissions, but looking at another working FreeBSD instance, I believe the permissions should be 600.

[Probotect0r]

I have figured out the solution. You need to create the empty file /firstboot in your packer provisioner. This file is used to determine whether the system is booting for the first time, and is removed at the end of the first boot process. There are some system services that only run on the first boot (i.e if the /firstboot file is present). Some of these are specifically for EC2 (located in /usr/local/etc/rc.d/), and one in particular is titled ec2_fetchkey, which seems to be fetching the SSH public key from the EC2 metadata API and adding it to the authorized_keys file. This service wasn't running when you launch the instance because the AMI created by Packer doesn't have the /firstboot file; the file gets removed when Packer launches the builder instance to run the provisioners.

I added this line to my shell provisioner:

su root -c 'touch /firstboot'