Hemant40 commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

[root@localhost Terraform]# terraform -v Terraform v1.2.7 on linux_amd64

provider registry.terraform.io/hashicorp/aws v4.27.0
provider registry.terraform.io/hashicorp/awscc v0.30.0

Affected Resource(s)

awscc_networkmanager_core_network
aws_route_table

Expected Behavior

After creating a Core network inside a global network the core_network_arn should be use into the aws_route_table because the static routes needs to pointed in the core_network.

Actual Behavior

Getting issue with the aws_route_table module, the created core_network and its ARN(core_network_arn) is not getting added to the route table.

And i think so that the issue is happening because the core_network_arn is read only once created

Steps to Reproduce

GLOBAL NETWORK - Optionally created if it is not supplied as variable

resource "awscc_networkmanager_global_network" "global_network" {
  count = var.create_global_network ? 1 : 0
  description = var.global_network.description

  tags = module.tags.tags
}

CORE NETWORK

resource "awscc_networkmanager_core_network" "core_network" {
  description       = var.core_network.description
  global_network_id = var.create_global_network ? awscc_networkmanager_global_network.global_network[0].id : var.global_network.id
  policy_document        = data.aws_networkmanager_core_network_policy_document.main.json

  tags = module.tags.tags
}

module "tags" {
  source  = "aws-ia/label/aws"
  version = "0.0.5"

  tags = var.tags
}

resource "awscc_networkmanager_vpc_attachment" "vpc_attachment" {
  subnet_arns             = [ aws_subnet.public-subnet.arn ]
  core_network_id         = awscc_networkmanager_core_network.core_network.id
  vpc_arn                 = aws_vpc.vpc.arn
  options                 = {
    ipv_6_support =  false
  }
  tags                    = [
    { key = "Name", value = "vpc"},
    { key = "segment", value = "shared"}
  ]
}

resource "aws_networkmanager_attachment_accepter" "vpc_acc" {
   attachment_id   = awscc_networkmanager_vpc_attachment.vpc_attachment.id
   attachment_type = awscc_networkmanager_vpc_attachment.vpc_attachment.attachment_type
}

resource "aws_route_table" "abctest" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "172.16.108.0/28"
    core_network_arn = awscc_networkmanager_core_network.core_network.core_network_arn
  }

  tags = {
    Name = "test"
  }
}

Adding the error snapshot here for better understanding.

ewbankkit commented 2 years ago

I can reproduce this:

provider "aws" {
  region = "us-west-2"
}

data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

resource "aws_vpc" "test" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "ewbankkit-test"
  }
}

resource "aws_subnet" "test" {
  count = 2

  vpc_id            = aws_vpc.test.id
  availability_zone = data.aws_availability_zones.available.names[count.index]
  cidr_block        = cidrsubnet(aws_vpc.test.cidr_block, 8, count.index)

  tags = {
    Name = "ewbankkit-test"
  }
}

resource "aws_route_table" "test" {
  vpc_id = aws_vpc.test.id

  route {
    cidr_block       = "172.16.108.0/28"
    core_network_arn = awscc_networkmanager_core_network.test.core_network_arn
  }

  tags = {
    Name = "ewbankkit-test"
  }
}

data "aws_region" "current" {}

resource "aws_networkmanager_global_network" "test" {
  tags = {
    Name = "ewbankkit-test"
  }
}

resource "awscc_networkmanager_core_network" "test" {
  global_network_id = aws_networkmanager_global_network.test.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.test.json))
}

data "aws_networkmanager_core_network_policy_document" "test" {
  core_network_configuration {
    # Don't overlap with default TGW ASN: 64512.
    asn_ranges = ["65022-65534"]

    edge_locations {
      location = data.aws_region.current.name
    }
  }

  segments {
    name = "test"
  }
}

% terraform apply
...
aws_networkmanager_global_network.test: Creating...
aws_vpc.test: Creating...
aws_vpc.test: Creation complete after 2s [id=vpc-0cb901d180584b730]
aws_subnet.test[0]: Creating...
aws_subnet.test[1]: Creating...
aws_networkmanager_global_network.test: Creation complete after 2s [id=global-network-017d6bc2b329a2ad6]
aws_subnet.test[1]: Creation complete after 0s [id=subnet-043352eca7fdfecd4]
awscc_networkmanager_core_network.test: Creating...
aws_subnet.test[0]: Creation complete after 1s [id=subnet-005ef3b3ab9cb6c3d]
awscc_networkmanager_core_network.test: Still creating... [10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [5m0s elapsed]
awscc_networkmanager_core_network.test: Creation complete after 5m6s [id=core-network-0bca1c657c9da8e01]
aws_route_table.test: Creating...
╷
│ Error: error creating Route in Route Table (rtb-067d0a8d46982eeb6) with destination (172.16.108.0/28): InvalidCoreNetworkArn.NotFound: The core network arn 'arn:aws:networkmanager::123456789012:core-network/core-network-0bca1c657c9da8e01' does not exist.
│   status code: 400, request id: 09a5a6f5-b883-44d5-9124-2398c04913ba
│ 
│   with aws_route_table.test,
│   on main.tf line 34, in resource "aws_route_table" "test":
│   34: resource "aws_route_table" "test" {
│ 
╵

ewbankkit commented 2 years ago

However, if I create a core network VPC attachment and then make the route table depend on that, it is created successfully:

provider "aws" {
  region = "us-west-2"
}

data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

resource "aws_vpc" "test" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "ewbankkit-test"
  }
}

resource "aws_subnet" "test" {
  count = 2

  vpc_id            = aws_vpc.test.id
  availability_zone = data.aws_availability_zones.available.names[count.index]
  cidr_block        = cidrsubnet(aws_vpc.test.cidr_block, 8, count.index)

  tags = {
    Name = "ewbankkit-test"
  }
}

resource "aws_route_table" "test" {
  vpc_id = aws_vpc.test.id

  route {
    cidr_block       = "172.16.108.0/28"
    core_network_arn = awscc_networkmanager_core_network.test.core_network_arn
  }

  tags = {
    Name = "ewbankkit-test"
  }

  depends_on = [aws_networkmanager_vpc_attachment.test]
}

data "aws_region" "current" {}

resource "aws_networkmanager_global_network" "test" {
  tags = {
    Name = "ewbankkit-test"
  }
}

resource "awscc_networkmanager_core_network" "test" {
  global_network_id = aws_networkmanager_global_network.test.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.test.json))
}

data "aws_networkmanager_core_network_policy_document" "test" {
  core_network_configuration {
    # Don't overlap with default TGW ASN: 64512.
    asn_ranges = ["65022-65534"]

    edge_locations {
      location = data.aws_region.current.name
    }
  }

  segments {
    name = "test"
  }
}

resource "aws_networkmanager_vpc_attachment" "test" {
  subnet_arns     = aws_subnet.test[*].arn
  core_network_id = awscc_networkmanager_core_network.test.id
  vpc_arn         = aws_vpc.test.arn

  tags = {
    Name = "ewbankkit-test"
  }
}

% terraform apply
...
aws_networkmanager_global_network.test: Creating...
aws_vpc.test: Creating...
aws_networkmanager_global_network.test: Creation complete after 2s [id=global-network-08917c1a9cab47fc4]
awscc_networkmanager_core_network.test: Creating...
aws_vpc.test: Creation complete after 2s [id=vpc-04356683940d4b179]
aws_subnet.test[0]: Creating...
aws_subnet.test[1]: Creating...
aws_subnet.test[0]: Creation complete after 1s [id=subnet-0c871c060f1d22edd]
aws_subnet.test[1]: Creation complete after 1s [id=subnet-0aa3bff617c95b58d]
awscc_networkmanager_core_network.test: Still creating... [10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [1m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [2m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m40s elapsed]
awscc_networkmanager_core_network.test: Still creating... [3m50s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m0s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m10s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m20s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m30s elapsed]
awscc_networkmanager_core_network.test: Still creating... [4m40s elapsed]
awscc_networkmanager_core_network.test: Creation complete after 4m45s [id=core-network-0cbea1e2221500894]
aws_networkmanager_vpc_attachment.test: Creating...
aws_networkmanager_vpc_attachment.test: Still creating... [10s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [20s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [30s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [40s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [50s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m0s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m10s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m20s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m30s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m40s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [1m50s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [2m0s elapsed]
aws_networkmanager_vpc_attachment.test: Still creating... [2m10s elapsed]
aws_networkmanager_vpc_attachment.test: Creation complete after 2m11s [id=attachment-0c113e5453b2e877c]
aws_route_table.test: Creating...
aws_route_table.test: Creation complete after 1s [id=rtb-0afe92edb78690c3b]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

ewbankkit commented 2 years ago

@Hemant40 Could you please add an explicit dependency on the VPC attachment (or associated accepter):

resource "awscc_networkmanager_core_network" "core_network" {
  description       = var.core_network.description
  global_network_id = var.create_global_network ? awscc_networkmanager_global_network.global_network[0].id : var.global_network.id
  policy_document        = data.aws_networkmanager_core_network_policy_document.main.json

  tags = module.tags.tags
}

module "tags" {
  source  = "aws-ia/label/aws"
  version = "0.0.5"

  tags = var.tags
}

resource "awscc_networkmanager_vpc_attachment" "vpc_attachment" {
  subnet_arns             = [ aws_subnet.public-subnet.arn ]
  core_network_id         = awscc_networkmanager_core_network.core_network.id
  vpc_arn                 = aws_vpc.vpc.arn
  options                 = {
    ipv_6_support =  false
  }
  tags                    = [
    { key = "Name", value = "vpc"},
    { key = "segment", value = "shared"}
  ]
}

resource "aws_networkmanager_attachment_accepter" "vpc_acc" {
   attachment_id   = awscc_networkmanager_vpc_attachment.vpc_attachment.id
   attachment_type = awscc_networkmanager_vpc_attachment.vpc_attachment.attachment_type
}

resource "aws_route_table" "abctest" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "172.16.108.0/28"
    core_network_arn = awscc_networkmanager_core_network.core_network.core_network_arn
  }

  tags = {
    Name = "test"
  }

  depends_on = [aws_networkmanager_attachment.vpc_attachment]
}

Hemant40 commented 2 years ago

Hi @ewbankkit, Thanks for sharing this information. I have checked my code as you suggested, But i have a strange behaviour with modules "awscc_networkmanager_vpc_attachment" & "aws_networkmanager_vpc_attachment".

If i use "aws_networkmanager_vpc_attachment" and add it as a dependency in "aws_route_table" everything is working fine.

But if i use "awscc_networkmanager_vpc_attachment" and add it as a dependency in "aws_route_table" it is still giving the same error i had shown in my snip above.

So i still think there is some issue with "awscc_networkmanager_vpc_attachment", Could you please check this behaviour once?

resource "awscc_networkmanager_core_network" "core_network" {
  description       = var.core_network.description
  global_network_id = var.create_global_network ? awscc_networkmanager_global_network.global_network[0].id : var.global_network.id
  policy_document        = data.aws_networkmanager_core_network_policy_document.main.json

  tags = module.tags.tags
}

module "tags" {
  source  = "aws-ia/label/aws"
  version = "0.0.5"

  tags = var.tags
}

resource "awscc_networkmanager_vpc_attachment" "vpc_attachment" {
  subnet_arns             = [ aws_subnet.public-subnet.arn ]
  core_network_id         = awscc_networkmanager_core_network.core_network.id
  vpc_arn                 = aws_vpc.vpc.arn
  options                 = {
    ipv_6_support =  false
  }
  tags                    = [
    { key = "Name", value = "vpc"},
    { key = "segment", value = "shared"}
  ]
}

resource "aws_networkmanager_attachment_accepter" "vpc_acc" {
   attachment_id   = awscc_networkmanager_vpc_attachment.vpc_attachment.id
   attachment_type = awscc_networkmanager_vpc_attachment.vpc_attachment.attachment_type
}

resource "aws_route_table" "abctest" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "172.16.108.0/28"
    core_network_arn = awscc_networkmanager_core_network.core_network.core_network_arn
  }

  tags = {
    Name = "test"
  }

  depends_on = [awscc_networkmanager_vpc_attachment.vpc_attachment]
}

hashicorp / terraform-provider-awscc

Core_network_arn issue in the VPC route table #637

Community Note

Terraform CLI and Terraform AWS Cloud Control Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Steps to Reproduce

GLOBAL NETWORK - Optionally created if it is not supplied as variable

CORE NETWORK