ResourceNotFound when Creating Amazon CloudWatch Dashboard

QI-D commented 2 years ago

name: 🐛 Bug Report about: I was creating a deployment and terraform returned a "ResourceNotFound" error message complaining about Amazon CloudWatch Dashboard not found when it's creating the dashboard with customized dashboard body.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform CLI Version: 1.2.5 AWS Provider Version: v4.13.0

Affected Resource(s)

aws_cloudwatch_dashboard

Terraform Configuration Files

provider "aws" {
  shared_credentials_files = ["/home/qdang/aws/aws-credentials.txt"]
  region  = "us-east-1"
}

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.13.0"
    }
    template = {
      source = "hashicorp/template"
    }
  }
}

resource "aws_vpc" "some_custom_vpc" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "Some Custom VPC"
  }
}

resource "aws_subnet" "some_public_subnet" {
  vpc_id            = aws_vpc.some_custom_vpc.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a"

  tags = {
    Name = "Some Public Subnet"
  }
}

resource "aws_subnet" "some_private_subnet" {
  vpc_id            = aws_vpc.some_custom_vpc.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-1a"

  tags = {
    Name = "Some Private Subnet"
  }
}

resource "aws_internet_gateway" "some_ig" {
  vpc_id = aws_vpc.some_custom_vpc.id

  tags = {
    Name = "Some Internet Gateway"
  }
}

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.some_custom_vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.some_ig.id
  }

  tags = {
    Name = "Public Route Table"
  }
}

resource "aws_route_table_association" "public_1_rt_a" {
  subnet_id      = aws_subnet.some_public_subnet.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_security_group" "some_sg" {
  name   = "HTTP and SSH"
  vpc_id = aws_vpc.some_custom_vpc.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_cloudwatch_log_group" "instance-log-group" {
  name = "qd-test"
}

resource "time_sleep" "delay_destroy_log_group" {
  depends_on = [aws_cloudwatch_log_group.instance-log-group]

  destroy_duration = "5s"
}

resource "aws_instance" "some_instance" {
  ami           = "ami-00e87074e52e6c9f9"
  instance_type = "t2.nano"
  key_name      = "qd-east1"

  subnet_id                   = aws_subnet.some_public_subnet.id
  vpc_security_group_ids      = [aws_security_group.some_sg.id]
  associate_public_ip_address = true

  tags = {
    "Name" : "qd-test"
  }
}

resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "some-dashboard"

  dashboard_body = <<EOF
{
    "widgets": [
        {
            "type": "log",
            "x": 0,
            "y": 0,
            "width": 12,
            "height": 6,
            "properties": {
                "region": "us-east-1",
                "title": "Latency (ms)",
                "query": "SOURCE 'qd-test' | filter @message like /round trip/ | parse @message 'Tx thread info: round trip time (ms) = *, variance' as rtt | stats pct(rtt, 50), pct(rtt, 80), pct(rtt, 90) by bin(10m)",
                "view": "timeSeries"
            }
        }
    ]
}
EOF
}

Debug Output

Not able to provide a complete debug output since this issue happens rarely and randomly, haven't been able to reproduce the issue when having TF_LOG set to DEBUG. Please see the following error message:

Error: reading Amazon CloudWatch Dashboard (hbxve-swin-0): ResourceNotFound: Dashboard hbxve-swin-0 does not exist
        status code: 404, request id: e0aea221-4441-4761-b437-3715549d9f02

        with module.win-std.aws_cloudwatch_dashboard.main[0],
        on ../../../modules/aws/win-std/main.tf line 200, in resource "aws_cloudwatch_dashboard" "main":
       200: resource "aws_cloudwatch_dashboard" "main" {

Expected Behavior

Expected Terraform to create AWS CloudWatch dashboard with customized widgets.

Actual Behavior

Terraform returned an error with error message "ResourceNotFound".

Steps to Reproduce

terraform apply

justinretzolk commented 2 years ago

Hey @QI-D 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to look into this, can you update the issue description to include all of the information requested in the bug report template?

QI-D commented 2 years ago

Hi @justinretzolk, thanks for responding. I've updated the description based on the bug report template. Only I'm not able to provide a complete debug output since this issue happens rarely and randomly. It only happened twice for the past month in our daily tests. I haven't been able to reproduce the issue when having TF_LOG set to DEBUG. So I provided the error message instead.

jstrese commented 1 year ago

This is also happening to us. It worked properly a few times but it has been a few months. During a deployment today we ran into it.

sripke commented 1 year ago

Had the exact same issue today: Terraform CLI Version: 1.4.0 AWS Provider Version: 4.58.0 Workaround was to simply provide the dashboard-resource manually on INT environment. But I fear this will happen again during PROD deployment.

QI-D commented 1 year ago

aws-deployment.log This issue just happened again recently. Please see the Terraform output attached.

hashicorp / terraform-provider-aws