puppetlabs / puppetlabs-pecdm

Puppet Bolt driven fusion of puppetlabs/peadm and Terraform.
Apache License 2.0
14 stars 18 forks source link

pecdm::provision is not idempotent due to aws_key_pair attempted recreation #97

Closed jessereynolds closed 1 year ago

jessereynolds commented 1 year ago

Describe the Bug

pecdm::provision is not idempotent - running the plan again on an already provisioned aws infra (albeit with reaped instances) causes the following error:

% bolt plan run pecdm::provision --params @params.json
Starting: plan pecdm::provision
Input Puppet Enterprise console password now or accept default. [puppetlabs]:
Starting: plan pecdm::subplans::provision
Starting: task terraform::initialize on localhost
Finished: task terraform::initialize with 0 failures in 0.17 sec
Starting infrastructure provisioning for a standard deployment of Puppet Enterprise
Starting: plan terraform::apply
Starting: task terraform::apply on localhost
Finished: task terraform::apply with 1 failure in 34.67 sec
Finished: plan terraform::apply in 34.68 sec
Finished: plan pecdm::subplans::provision in 35.76 sec
Finished: plan pecdm::provision in 39.71 sec
Failed on localhost:

  Error: Error import KeyPair: InvalidKeyPair.Duplicate: The keypair 'pe_adm_14e8e3' already exists.
    status code: 400, request id: 257c2dd5-8d06-4e8c-902b-0dbe086c6f67

    with module.instances.aws_key_pair.pe_adm,
    on modules/instances/main.tf line 54, in resource "aws_key_pair" "pe_adm":
    54: resource "aws_key_pair" "pe_adm" {

Failed on 1 target: localhost
Ran on 1 target

Expected Behavior

I expect this plan to run the terraform apply again and have it create anything that currently does not exist.

Steps to Reproduce

Create params.json:

{
    "project"        : "jr-ape",
    "version"        : "2021.7.1",
    "architecture"   : "standard",
    "cluster_profile": "development",
    "provider"       : "aws",
    "cloud_region"   : "ap-southeast-2",
    "firewall_allow" : [ "127.0.0.1/32" ],
    "ssh_pub_key_file" : "/Users/jesse/.ssh/id_rsa.pub",
    "extra_terraform_vars" : {
    "tags" : {
      "lifetime" : "2d",
      "department" : "PS",
      "created_by" : "example.4hxuydm@perforce.com"
    }
  }
}

Sign in to aws with aws sso login

Execute the plan as per above: bolt plan run pecdm::provision --params @params.json

Delete the created ec2 instances.

Execute the plan again: bolt plan run pecdm::provision --params @params.json

Environment

Additional Context

Looks like aws_key_pair resource state is not being added to tfstate. I see there is this issue https://github.com/hashicorp/terraform-provider-aws/issues/1092 about aws_key_pair not being able to be idempotent unless you hack the tfstate file by adding in the public_key if it's not there already.

I also tried just deleting the key pair and running the provision plan again, but then it creates a second pe server instance (same name) and then goes on to pass invalid parameters to peadm::deploy (passing undef to primary_host)...

...
Starting: plan peadm::install
Finished: plan peadm::install in 0.03 sec
Finished: plan pecdm::subplans::deploy in 0.08 sec
Finished: plan pecdm::provision in 52.36 sec
peadm::install: parameter 'primary_host' expects a Peadm::SingleTargetSpec = Variant[Pattern[/\A[^[:space:],]+\z/], Object[{name => 'Target', attributes => {'uri' => {type => Optional[String[1]], kind => given_or_derived}, 'name' => {type => Optional[String[1]], kind => given_or_derived}, 'safe_name' => {type => Optional[String[1]], kind => given_or_derived}, 'target_alias' => {type => Optional[Variant[String[1], Array[String[1]]]], kind => given_or_derived}, 'config' => {type => Optional[Hash[String[1], Data]], kind => given_or_derived}, 'vars' => {type => Optional[Hash[String[1], Data]], kind => given_or_derived}, 'facts' => {type => Optional[Hash[String[1], Data]], kind => given_or_derived}, 'features' => {type => Optional[Array[String[1]]], kind => given_or_derived}, 'plugin_hooks' => {type => Optional[Hash[String[1], Data]], kind => given_or_derived}, 'resources' => {type => Optional[Hash[String[1], ResourceInstance]], kind => given_or_derived}}, functions => {'host' => Callable[[0, 0], Optional[String]], 'password' => Callable[[0, 0], Optional[String[1]]], 'port' => Callable[[0, 0], Optional[Integer]], 'protocol' => Callable[[0, 0], Optional[String[1]]], 'transport' => Callable[[0, 0], String[1]], 'transport_config' => Callable[[0, 0], Hash[String[1], Data]], 'user' => Callable[[0, 0], Optional[String[1]]]}}], Array[Peadm::SingleTargetSpec, 1, 1]] value, got Undef
  (file: /Users/jesse/src/puppet/tf/puppetlabs-pecdm/plans/subplans/deploy.pp, line: 70, column: 3)
jessereynolds commented 1 year ago

I'm pretty sure this is only happening in relation to #99 and I don't think there's much else to be done to make this more resilient.