S3 bootstrapping using ASG module is not working

[jschelfh-be]

Describe the bug

Trying to deploy the vmseries FW using the asg module. Placed the bootstrap.xml and init-cfg.txt under config folder in an S3 bucket. The FW deploys fine, but the bootstrapping is not happening. Important to mention that this is working fine when using vmseries module - so I'm confident the structure of the s3 bucket is ok.

Expected behavior

proper bootstrapping on the FW deployment when using S3 bucket

Current behavior

no bootstrapping being done I noticed in the asg module that the parameter build-up passed in the user_data is slightly different. the asg-module uses a newline while in the vmseries example a semi-colon is used as seperator.

vmseries-bootstrap-aws-s3bucket=<bucket_name>
mgmt-interface-swap=enable
op-cmd-dpdk-pkt-io=on

vs

vmseries-bootstrap-aws-s3bucket=<bucket_name>;mgmt-interface-swap=enable;op-cmd-dpdk-pkt-io=on

I have tried both - same result

Steps to reproduce

locals {
  vmseries_interfaces = {
    data = {
      device_index   = 0
      security_group = "vmseries_data"
      subnet = {
        "eu-central-1a" = "data",
        "eu-central-1b" = "data"        
      }
      create_public_ip  = false
      source_dest_check = false
    }
    mgmt = {
      device_index   = 1
      security_group = "vmseries_mgmt"
      subnet = {
        "eu-central-1a" = "mgmt",
        "eu-central-1b" = "mgmt"        
      }
      create_public_ip  = true
      source_dest_check = true
    }
  }

  bootstrap_options = {
    vmseries-bootstrap-aws-s3bucket = module.bootstrap.bucket_name
    op-cmd-dpdk-pkt-io  = "on"
    mgmt-interface-swap = "enable"
  }
}

# ======================================================
# Create S3 bucket and upload bootstrap config
# ======================================================
module "bootstrap" {
  source  = "PaloAltoNetworks/vmseries-modules/aws//modules/bootstrap"
  version = "0.4.1"

  prefix                = var.name_prefix
  bucket_name           = var.vmseries_bucket_name
  force_destroy         = false
  source_root_directory = var.vmseries_source_root_directory
}

# ======================================================
# Create FireWall vmseries instances (ASG)
# ======================================================
module "vmseries_asg" {
  source  = "PaloAltoNetworks/vmseries-modules/aws//modules/asg"
  version = "0.4.1"

  asg_name              = "vmseries"
  name_prefix           = "${var.name_prefix}-"
  vmseries_product_code = "6njl1pau431dv1qxipg63mvah"
  vmseries_version      = "10.2.2"
  bootstrap_options     = local.bootstrap_options
  ssh_key_name          = var.vmseries_ssh_key_name
  fw_license_type       = "byol"

  desired_capacity  = 2
  min_size          = 2
  max_size          = 4

  interfaces = {
    for k, v in local.vmseries_interfaces : k => {
      device_index       = v.device_index
      security_group_ids = try([module.security_vpc.security_group_ids[v.security_group]], [])
      source_dest_check  = try(v.source_dest_check, false)
      subnet_id          = { for z, s in v.subnet : z => module.security_subnet_sets[s].subnets[z].id }      
      create_public_ip   = try(v.create_public_ip, false)
    }
  }
}

Your Environment

Terraform v1.3.6 on linux_amd64

• provider registry.terraform.io/hashicorp/archive v2.3.0
• provider registry.terraform.io/hashicorp/aws v4.51.0
• provider registry.terraform.io/hashicorp/random v3.3.2
• provider registry.terraform.io/infobloxopen/infoblox v2.2.0

[welcome-to-palo-alto-networks[bot]]

:tada: Thanks for opening your first issue here! Welcome to the community!

[jschelfh-be]

The parameters passed in the user data of the launch_template do not seem to get picked-up. If I create a launch_template from an existing (working) ec2-vmseries, it does not work either creating a new instance from the template.

[jschelfh-be]

I figured out what the problem is. In the logs you can see 2023-02-01T10:41:55.118+01:00 ERROR : BOOTSTRAP : FAIL : 1675244515118 : Media Detection - Failed - No bootstrap media detected.. 2023-02-01T10:42:01.795+01:00 INFO : MGMTINTERFACE : COMPLETE : Firewall Interface Swap Configured. 2023-02-01T10:43:48.663+01:00 INFO : PANOS : START : Firewall version 10.2.2 starting.

The mgmt interface gets setup through the lambda. For the S3 (or Secret manager) to be accessible the mgmt interface needs to be active. If I attach for example an elastic IP immediately after the instance is launched from the template then the bootstrapping is working fine. I think you need to make the mgmt interface configuration part of the launch template...

[jschelfh-be]

piggypacked on the change made in #283 added instance profile to launch template to get bootstrapping through S3/Secrets Manager to work #281

[migara]

Good catch, please let us know if there are any more issues with bootstrapping after setting the instance_profile