Chef Provisioner - Invalid Private Key

[ghost]

This issue was originally opened by @BMonsalvatge as hashicorp/terraform#18461. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

resource "aws_instance" "bastion_server" {
  ami                    = "ami-5cc39523"
  instance_type          = "t2.micro"
  subnet_id              = "${data.terraform_remote_state.core.public_subnets[0]}"
  vpc_security_group_ids = ["${module.bastion_sg.this_security_group_id}", "${data.terraform_remote_state.core.test_sg}"]
  key_name               = "${data.terraform_remote_state.core.central_key_pair}"

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = "${file(var.provisioner_key)}"
    agent       = false
  }

  provisioner "chef" {
    environment     = "utility"
    run_list        = ["cookbook::bastion"]
    node_name       = "${aws_instance.bastion_server.tags.Name}"
    server_url      = "https://chef.server.com/organizations/orgname/"
    user_name       = "${var.chef_user}"
    user_key        = "${var.chef_key}"
    ssl_verify_mode = ":verify_peer"
    version         = "${var.chef_version}"
    recreate_client = true
  }
  tags {
    Name         = "bastion_server"
  }
}

Expected Behavior

Chef should have bootstrapped the node.

Actual Behavior

aws_instance.bastion_server (chef): Preparing to unpack .../chef_14.3.37-1_amd64.deb ...
aws_instance.bastion_server (chef): Unpacking chef (14.3.37-1) ...
aws_instance.bastion_server: Still creating... (40s elapsed)
aws_instance.bastion_server (chef): Setting up chef (14.3.37-1) ...
aws_instance.bastion_server (chef): Thank you for installing Chef!
aws_instance.bastion_server (chef): Creating configuration files...
aws_instance.bastion_server (chef): Generate the private key...
aws_instance.bastion_server: Still creating... (50s elapsed)
aws_instance.bastion_server (chef): Cleanup user key...
aws_instance.bastion_server (chef): ERROR: Chef::Exceptions::InvalidPrivateKey: The file /etc/chef/validator.pem or :raw_key option does not contain a correctly formatted private key or the key is encrypted.
aws_instance.bastion_server (chef): The key file should begin with '-----BEGIN RSA PRIVATE KEY-----' and end with '-----END RSA PRIVATE KEY-----'
Releasing state lock. This may take a few moments...

Additionally on the server the contents of /etc/chef/ are the following:

-rw-------  1 root root  192 Jul 15 04:44 client.rb
-rw-------  1 root root   37 Jul 15 04:44 first-boot.json

contents of client.rb are:

log_location            STDOUT
chef_server_url         "https://chef.server.com/organizations/orgname/"
node_name               "bastion_server

ssl_verify_mode  :verify_peer

If I add the correct key to /etc/chef/validator.pem & edit the client.rb file to look like the following, sudo chef-client works and connects to the chef server:

log_location            STDOUT
chef_server_url         "https://chef.server.com/organizations/orgname/"
node_name               "bastion_server

validation_client_name   'validator'
validation_key      '/etc/chef/validator.pem'

ssl_verify_mode  :verify_peer

Steps to Reproduce

terraform init terraform apply

Additional Context

References

[BMonsalvatge]

In case anyone else is experiencing similar issues, for now my workaround has been running local-exec provisioners:

# Sleep because local-exec was running before ssh was started newly created instances.
provisioner "local-exec" {
    command = "sleep 10s"
  }
  provisioner "local-exec" {
    command = "cd ${var.knife_path} && knife node delete ${aws_instance.bastion_server.tags.Name} -y && knife client delete ${aws_instance.bastion_server.tags.Name} -y || true"
  }
  provisioner "local-exec" {
    command = "cd ${var.knife_path} && knife bootstrap ${aws_instance.bastion_server.public_ip} -N ${aws_instance.bastion_server.tags.Name} -r 'role[bastion]' -x ubuntu -i ${var.provisioner_key} --sudo"
  }
  provisioner "local-exec" {
    command = "cd ${var.knife_path} && knife node run_list add ${aws_instance.bastion_server.tags.Name} 'role[bastion]'"
  }

[stobias123]

hey @BMonsalvatge - I was experiencing this same issue, but then discovered it was because I was putting the path to my private key in improperly.

Compare this

variable "chef_key" {
  default = "/path/to/file.pem"
}

...

user_key        = "${var.chef_key}"

With this

user_key        = "${file("/path/to/file.pem")}"

The file interpolation is returning the contents of the pem key as a string, while the variable is just returning the path. You must use the file interpolation, or I assume you can even post the whole private key directly if you really want to....

https://www.terraform.io/docs/configuration/interpolation.html#file-path-

[BMonsalvatge]

Hey @stobias123 - Not sure how I missed that one considering I put it in the connection block above. Thanks for pointing that out! Resolved my issue.