Provider produced inconsistent result after apply when creating firewall rules

[deigaard]

Preliminary checks

• [X] I am using the latest version, or the latest version that corresponds to my Oxide installation.
• [X] There is no open issue that reports the same problem.

What was the expected behaviour

Per #219, when adding new firewall rules to a new, non-default VPC using terraform, I'm expecting that I need to re-create the existing, default firewall rules with a new terraform-provided copy of the default ones so that I can add some additional firewall rules.

In the end, I'm expecting to have the default rules plus the ones I add.

What is the current behaviour and what actions did you take to get there

When I attempt to do this with an already created VPC named globus, I'm getting the following results:

• a mostly clean terraform run
• a provisioned VM
• a correct looking set of firewall rules (defaults plus one new)

However, I get an error for each firewall rule that says:

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to oxide_vpc_firewall_rules.globus, provider
│ "provider[\"registry.terraform.io/oxidecomputer/oxide\"]" produced an unexpected new value: .rules:
│ planned set element cty.ObjectVal(map[string]cty.Value{"action":cty.StringVal("allow"),
│ "description":cty.StringVal("Inbound"), "direction":cty.StringVal("inbound"),
│ "filters":cty.ObjectVal(map[string]cty.Value{"hosts":cty.NullVal(cty.Set(cty.Object(map[string]cty.Type{"type":cty.String,
│ "value":cty.String}))), "ports":cty.NullVal(cty.Set(cty.String)),
│ "protocols":cty.NullVal(cty.Set(cty.String))}), "id":cty.UnknownVal(cty.String),
│ "name":cty.StringVal("allow-internal-inbound"), "priority":cty.NumberIntVal(65534),
│ "status":cty.StringVal("enabled"),
│ "targets":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"type":cty.StringVal("vpc"),
│ "value":cty.StringVal("globus")})}), "time_created":cty.UnknownVal(cty.String),
│ "time_modified":cty.UnknownVal(cty.String)}) does not correlate with any element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

The terraform code that I'm using is:

terraform {
  required_version = ">= 1.0"

  required_providers {
    oxide = {
      source  = "oxidecomputer/oxide"
      version = ">= 0.3.0"
    }
  }
}

data "oxide_instance_external_ips" "globus-1" {
  instance_id = oxide_instance.globus-1.id
}

output "external_ips" {
  value = data.oxide_instance_external_ips.globus-1.external_ips[0].ip
}

resource "oxide_ssh_key" "soren" {
  name        = "soren-ssh-ed25519"
  description = "soren-ssh-ed25519"
  public_key  = "<redacted, but not really necessary to do so>"
}

data "oxide_project" "project" {
  name = "globus"
  timeouts = {
    read = "1m"
  }
}

resource "oxide_vpc" "vpc" {
  project_id  = data.oxide_project.project.id
  description = "globus vpc"
  name        = "globus"
  dns_name    = "globus"
  #ipv6_prefix = "fd1e:4947:d4a1::/48"
  timeouts = {
    read   = "1m"
    create = "3m"
    delete = "2m"
    update = "2m"
  }
}

resource "oxide_vpc_subnet" "subnet" {
  vpc_id      = oxide_vpc.vpc.id
  description = "Globus vpc subnet"
  name        = "globus"
  ipv4_block  = "172.31.0.0/22"
  #ipv6_block  = "fdaf:27b3:c28c::/64"
  timeouts = {
    read   = "1m"
    create = "3m"
    delete = "2m"
    update = "2m"
  }
}

resource "oxide_vpc_firewall_rules" "globus" {
  vpc_id = oxide_vpc.vpc.id
  rules = [
    {
      action      = "allow"
      description = "Inbound"
      name        = "custom-allow-globus"
      direction   = "inbound"
      priority    = 50
      status      = "enabled"
      filters = {
        ports     = ["50000-51000", "443"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "subnet"
          value = "globus"
        }
      ]
    },
    {
      action      = "allow"
      description = "Inbound"
      name        = "allow-ssh"
      direction   = "inbound"
      priority    = 65534
      status      = "enabled"
      filters = {
        ports     = ["22"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "vpc"
          value = "globus"
        }
      ]
    },
    {
      action      = "allow"
      description = "Inbound"
      name        = "allow-internal-inbound"
      direction   = "inbound"
      priority    = 65534
      status      = "enabled"
      filters = {
        type      = "vpc"
        value     = "globus"
      },
      targets = [
        {
          type  = "vpc"
          value = "globus"
        }
      ]
    },
    {
      action      = "allow"
      description = "Inbound "
      name        = "allow-icmp"
      direction   = "inbound"
      priority    = 65534
      status      = "enabled"
      filters = {
        protocols = ["ICMP"]
      },
      targets = [
        {
          type  = "vpc"
          value = "globus"
        }
      ]
    }
  ]
}

data "oxide_image" "image" {
  #project_name = "soren"      # Pulls image from silo
  name         = "rhel93"
  timeouts = {
    read = "1m"
  }
}

resource "oxide_disk" "globus-1" {
  project_id  = data.oxide_project.project.id
  description = "globus-1"
  name        = "globus-1"
  size        = 42949672960
  #block_size  = 512
  source_image_id = data.oxide_image.image.id

  timeouts = {
    read   = "1m"
    create = "3m"
    delete = "2m"
  }
}

resource "oxide_instance" "globus-1" {
  project_id      = data.oxide_project.project.id
  description     = "globus-1"
  name            = "globus-1"
  host_name       = "globus-1"
  memory          = 10737418240
  ncpus           = 4
  start_on_create = true
  ssh_public_keys = [ oxide_ssh_key.soren.id ]

  external_ips = [
    {
      ip_pool_name = "default"
      type         = "ephemeral"
    }
  ]

  disk_attachments = [ oxide_disk.globus-1.id ]
  network_interfaces = [
    {
      subnet_id   = oxide_vpc_subnet.subnet.id
      vpc_id      = oxide_vpc.vpc.id
      description = "globus-1-nic-0"
      name        = "globus-1-nic-0"
    },
  ]
  timeouts = {
    read   = "1m"
    create = "3m"
    delete = "2m"
  }
}

The terraform apply plan looks like this:

❯ terraform apply
data.oxide_project.project: Reading...
data.oxide_image.image: Reading...
data.oxide_project.project: Read complete after 0s [id=38b4d93a-3dde-4502-bf63-6ffb19f66e7d]
data.oxide_image.image: Read complete after 0s [id=65094592-4707-48f2-bf3d-f1c59a70bd26]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.oxide_instance_external_ips.globus-1 will be read during apply
  # (config refers to values not yet known)
 <= data "oxide_instance_external_ips" "globus-1" {
      + external_ips = (known after apply)
      + id           = (known after apply)
      + instance_id  = (known after apply)
    }

  # oxide_disk.globus-1 will be created
  + resource "oxide_disk" "globus-1" {
      + block_size      = (known after apply)
      + description     = "globus-1"
      + device_path     = (known after apply)
      + id              = (known after apply)
      + name            = "globus-1"
      + project_id      = "38b4d93a-3dde-4502-bf63-6ffb19f66e7d"
      + size            = 42949672960
      + source_image_id = "65094592-4707-48f2-bf3d-f1c59a70bd26"
      + time_created    = (known after apply)
      + time_modified   = (known after apply)
      + timeouts        = {
          + create = "3m"
          + delete = "2m"
          + read   = "1m"
        }
    }

  # oxide_instance.globus-1 will be created
  + resource "oxide_instance" "globus-1" {
      + description        = "globus-1"
      + disk_attachments   = [
          + (known after apply),
        ]
      + external_ips       = [
          + {
              + type = "ephemeral"
            },
        ]
      + host_name          = "globus-1"
      + id                 = (known after apply)
      + memory             = 10737418240
      + name               = "globus-1"
      + ncpus              = 4
      + network_interfaces = [
          + {
              + description   = "globus-1-nic-0"
              + id            = (known after apply)
              + ip_address    = (known after apply)
              + mac_address   = (known after apply)
              + name          = "globus-1-nic-0"
              + primary       = (known after apply)
              + subnet_id     = (known after apply)
              + time_created  = (known after apply)
              + time_modified = (known after apply)
              + vpc_id        = (known after apply)
            },
        ]
      + project_id         = "38b4d93a-3dde-4502-bf63-6ffb19f66e7d"
      + ssh_public_keys    = [
          + (known after apply),
        ]
      + start_on_create    = true
      + time_created       = (known after apply)
      + time_modified      = (known after apply)
      + timeouts           = {
          + create = "3m"
          + delete = "2m"
          + read   = "1m"
        }
    }

  # oxide_ssh_key.soren will be created
  + resource "oxide_ssh_key" "soren" {
      + description   = "soren-ssh-ed25519"
      + id            = (known after apply)
      + name          = "soren-ssh-ed25519"
      + public_key    = "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBACnzTEzYf1S06dshMkbm0AIUAh/szN9gQBq3ocW3T37MrnQ8u0BPw082JlHcX+NVjAFpwGx6YtpfbmHFT2479+nygDikc70auC4cs6eWl83g3hl8AaGoBxLr1ZfxDnqMCf+g/0GSuFkdAF3Tm5g8lEqIIaL0JX1merWCAhz9YYm5DTulw== soren@IT-G9XLG9255F"
      + silo_user_id  = (known after apply)
      + time_created  = (known after apply)
      + time_modified = (known after apply)
    }

  # oxide_vpc.vpc will be created
  + resource "oxide_vpc" "vpc" {
      + description      = "globus vpc"
      + dns_name         = "globus"
      + id               = (known after apply)
      + ipv6_prefix      = (known after apply)
      + name             = "globus"
      + project_id       = "38b4d93a-3dde-4502-bf63-6ffb19f66e7d"
      + system_router_id = (known after apply)
      + time_created     = (known after apply)
      + time_modified    = (known after apply)
      + timeouts         = {
          + create = "3m"
          + delete = "2m"
          + read   = "1m"
          + update = "2m"
        }
    }

  # oxide_vpc_firewall_rules.globus will be created
  + resource "oxide_vpc_firewall_rules" "globus" {
      + id     = (known after apply)
      + rules  = [
          + {
              + action        = "allow"
              + description   = "Inbound "
              + direction     = "inbound"
              + filters       = {
                  + protocols = [
                      + "ICMP",
                    ]
                }
              + id            = (known after apply)
              + name          = "allow-icmp"
              + priority      = 65534
              + status        = "enabled"
              + targets       = [
                  + {
                      + type  = "vpc"
                      + value = "globus"
                    },
                ]
              + time_created  = (known after apply)
              + time_modified = (known after apply)
            },
          + {
              + action        = "allow"
              + description   = "Inbound"
              + direction     = "inbound"
              + filters       = {
                  + ports     = [
                      + "22",
                    ]
                  + protocols = [
                      + "TCP",
                    ]
                }
              + id            = (known after apply)
              + name          = "allow-ssh"
              + priority      = 65534
              + status        = "enabled"
              + targets       = [
                  + {
                      + type  = "vpc"
                      + value = "globus"
                    },
                ]
              + time_created  = (known after apply)
              + time_modified = (known after apply)
            },
          + {
              + action        = "allow"
              + description   = "Inbound"
              + direction     = "inbound"
              + filters       = {
                  + ports     = [
                      + "443",
                      + "50000-51000",
                    ]
                  + protocols = [
                      + "TCP",
                    ]
                }
              + id            = (known after apply)
              + name          = "custom-allow-globus"
              + priority      = 50
              + status        = "enabled"
              + targets       = [
                  + {
                      + type  = "subnet"
                      + value = "globus"
                    },
                ]
              + time_created  = (known after apply)
              + time_modified = (known after apply)
            },
          + {
              + action        = "allow"
              + description   = "Inbound"
              + direction     = "inbound"
              + filters       = {}
              + id            = (known after apply)
              + name          = "allow-internal-inbound"
              + priority      = 65534
              + status        = "enabled"
              + targets       = [
                  + {
                      + type  = "vpc"
                      + value = "globus"
                    },
                ]
              + time_created  = (known after apply)
              + time_modified = (known after apply)
            },
        ]
      + vpc_id = (known after apply)
    }

  # oxide_vpc_subnet.subnet will be created
  + resource "oxide_vpc_subnet" "subnet" {
      + description   = "Globus vpc subnet"
      + id            = (known after apply)
      + ipv4_block    = "172.31.0.0/22"
      + ipv6_block    = (known after apply)
      + name          = "globus"
      + time_created  = (known after apply)
      + time_modified = (known after apply)
      + timeouts      = {
          + create = "3m"
          + delete = "2m"
          + read   = "1m"
          + update = "2m"
        }
      + vpc_id        = (known after apply)
    }

Plan: 6 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + external_ips = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

oxide_ssh_key.soren: Creating...
oxide_vpc.vpc: Creating...
oxide_disk.globus-1: Creating...
oxide_ssh_key.soren: Creation complete after 0s [id=9dbeed51-4278-4baf-8c03-ed2ee3212ed7]
oxide_vpc.vpc: Creation complete after 0s [id=6400237e-b396-4b4b-9594-b11f32031ed5]
oxide_vpc_subnet.subnet: Creating...
oxide_vpc_firewall_rules.globus: Creating...
oxide_vpc_subnet.subnet: Creation complete after 0s [id=066f01e5-3ae7-4eb6-91c8-48db0c4ea1fb]
oxide_disk.globus-1: Creation complete after 2s [id=3d3cadba-1e96-40d0-bec0-1782e10238f8]
oxide_instance.globus-1: Creating...
oxide_instance.globus-1: Creation complete after 9s [id=920bb86d-f443-4c86-9eb0-350dbd99ede4]
data.oxide_instance_external_ips.globus-1: Reading...
data.oxide_instance_external_ips.globus-1: Read complete after 0s [id=7e90f53f-32b2-4e59-883d-db04e642aacf]

Provider version

0.3.0

Terraform version

1.8.4

Operating system

darwin_arm64 - Mac

Anything else you would like to add?

No response

[karencfv]

Hi @deigaard, thanks for providing such a detailed bug report!

It looks like the Terraform state is reporting an empty array for hosts, protocols and ports, while your Terraform configuration is setting them as null. I'll work on a fix this week.

In the mean time, to suppress these annoying errors, you can set these values for each rule on the oxide_vpc_firewall_rules resource as an empty array on your configuration file where necessary, like this:

filters = {
  hosts     = []
  ports     = []
  protocols = []
},

[deigaard]

I gotta say, running terraform (or really anything with the Oxide system) is so fast that it was a pleasure to take a little bit of time to put together a decent, and repeatable bug summary.

Thank you for the workaround. It works great.

[karencfv]

Thanks for the kind words!

Fix is implemented, will be part of the next release. Thanks for helping us improve our product :)