[Bug]: Many provider crashes when running from GitLab CI pipeline #1944

Closed lsanchez-v closed 7 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Provider Version

v1.15.1

Terraform Version

v1.6.6

Terraform Edition

Terraform Open Source (OSS)

Current Behavior

When running Terraform from GitLab pipelines, I'm getting this provider related errors:

│ Warning: Deprecated Environment Variable
│ 
│   on providers.tf line 25, in provider "aws":
│   25: provider "aws" {
│ 
│ The environment variable "AWS_STS_ENDPOINT" is deprecated. Use environment
│ variable "AWS_ENDPOINT_URL_STS" instead.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for
│ mongodbatlas_advanced_cluster.atlas.accept_data_risks_and_force_replica_set_reconfig:
│ planned value cty.StringVal("") for a non-computed attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for mongodbatlas_advanced_cluster.atlas.bi_connector_config: block
│ count in plan (1) disagrees with count in config (0).
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for
│ mongodbatlas_advanced_cluster.atlas.replication_specs[0].num_shards:
│ planned value cty.NumberIntVal(1) for a non-computed attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for
│ mongodbatlas_advanced_cluster.atlas.replication_specs[0].zone_name: planned
│ value cty.StringVal("ZoneName managed by Terraform") for a non-computed
│ attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for
│ mongodbatlas_advanced_cluster.atlas.replication_specs[0].region_configs[0].backing_provider_name:
│ planned value cty.StringVal("") for a non-computed attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced invalid plan
│ 
│ Provider "registry.terraform.io/mongodb/mongodbatlas" planned an invalid
│ value for
│ mongodbatlas_advanced_cluster.atlas.replication_specs[0].region_configs[0].electable_specs[0].ebs_volume_type:
│ planned value cty.StringVal("") for a non-computed attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

### Terraform configuration to reproduce the issue

```shell
resource "mongodbatlas_advanced_cluster" "atlas" {
  project_id                  = var.mongodb_atlas_project_id
  name                        = var.mongodb_atlas_cluster_name
  mongo_db_major_version      = var.mongodb_major_version
  cluster_type                = "REPLICASET"
  encryption_at_rest_provider = var.mongodb_atlas_cluster_encryption_enabled ? "AWS" : "NONE"
  backup_enabled              = var.mongodb_atlas_cluster_enable_backup
  retain_backups_enabled      = var.mongodb_atlas_cluster_retain_backups
  pit_enabled                 = var.mongodb_atlas_cluster_enable_backup

  replication_specs {
    region_configs {
      electable_specs {
        instance_size = var.provider_instance_size_name
        node_count    = var.mongodb_replicaset_node_count
      }
      auto_scaling {
        disk_gb_enabled = true
      }
      provider_name = "AWS"
      region_name   = var.provider_region_name
      priority      = 7
    }
  }

  depends_on = [mongodbatlas_privatelink_endpoint_service.mongo_pl_endpoint_service]
  lifecycle {
    ignore_changes = [disk_size_gb]
  }
}

resource "mongodbatlas_cloud_backup_schedule" "atlas_snapshot_schedule" {
  count        = var.mongodb_atlas_cluster_enable_backup ? 1 : 0
  cluster_name = mongodbatlas_advanced_cluster.atlas.name
  project_id   = var.mongodb_atlas_project_id

  reference_hour_of_day    = 13
  reference_minute_of_hour = 45
  restore_window_days      = 7

  policy_item_hourly {
    frequency_interval = 6
    retention_unit     = "days"
    retention_value    = 7
  }
  policy_item_daily {
    frequency_interval = 1 # every 1 day
    retention_unit     = "days"
    retention_value    = 7
  }
  policy_item_weekly {
    frequency_interval = 6
    retention_unit     = "weeks"
    retention_value    = 4
  }
  policy_item_monthly {
    frequency_interval = 40 # every last day of the month
    retention_unit     = "months"
    retention_value    = 12
  }
}

#
# Configure routing to Atlas within the AWS cluster
#
resource "aws_vpc_security_group_egress_rule" "mongo_atlas" {
  for_each          = toset(var.aws_security_groups)
  security_group_id = each.value
  cidr_ipv4         = var.mongodb_atlas_cidr_block
  from_port         = 27015
  to_port           = 27017
  ip_protocol       = "tcp"
  description       = "Outbound connections to MongoDB Atlas"
}

data "aws_network_interface" "mongo_atlas_private_endpoint_interfaces" {
  for_each = aws_vpc_endpoint.mongo_endpoint.network_interface_ids
  id       = each.value
}

resource "aws_vpc_security_group_egress_rule" "mongo_atlas_private_endpoint" {
  for_each = merge([
    for sg in var.aws_security_groups : {
      for pe_ip in data.aws_network_interface.mongo_atlas_private_endpoint_interfaces :
      "${sg}-${pe_ip.private_ip}" => {
        sg_id = sg
        pe_ip = pe_ip.private_ip
      }
    }
  ]...)

  security_group_id = each.value.sg_id
  cidr_ipv4         = "${each.value.pe_ip}/32"
  from_port         = 0
  to_port           = 65535
  ip_protocol       = "tcp"
  description       = "Outbound connections to MongoDB Atlas Private Endpoint"
}

data "aws_subnets" "live_vpc_eks_private_subnets" {
  filter {
    name   = "vpc-id"
    values = [var.aws_vpc_id]
  }
  filter {
    name   = "tag:Name"
    values = [var.aws_vpc_subnet_name]
  }
}

data "aws_subnets" "app_subnets" {
  filter {
    name   = "vpc-id"
    values = [var.aws_vpc_id]
  }

  filter {
    name   = "tag:Name"
    values = [var.aws_vpc_subnet_name]
  }
}

resource "aws_security_group" "mongo_endpoint_sg" {
  name        = "${var.project}-${var.env}-mongo-atlas-ie-sg"
  description = "Security group fo the Mongo Atlas Interface Endpoint"
  vpc_id      = var.aws_vpc_id
  ingress {
    security_groups = var.aws_security_groups
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

data "aws_ssm_parameter" "mongodb_privatelink_data" {
  for_each = toset(["endpoint_service_name", "project_id", "private_link_id"])
  name     = "/mongodb_atlas/private_link/${each.value}"
}

resource "aws_vpc_endpoint" "mongo_endpoint" {
  vpc_id             = var.aws_vpc_id
  service_name       = data.aws_ssm_parameter.mongodb_privatelink_data["endpoint_service_name"].value
  vpc_endpoint_type  = "Interface"
  subnet_ids         = data.aws_subnets.app_subnets.ids
  security_group_ids = [aws_security_group.mongo_endpoint_sg.id]
}

resource "mongodbatlas_privatelink_endpoint_service" "mongo_pl_endpoint_service" {
  project_id          = data.aws_ssm_parameter.mongodb_privatelink_data["project_id"].value
  private_link_id     = data.aws_ssm_parameter.mongodb_privatelink_data["private_link_id"].value
  endpoint_service_id = aws_vpc_endpoint.mongo_endpoint.id
  provider_name       = "AWS"
}

Steps To Reproduce

terraform plan

Logs

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks for opening this issue! Please make sure you've followed our guidelines when opening the issue. In short, to help us reproduce the issue we need:

Terraform configuration file used to reproduce the issue
Terraform log files from the run where the issue occurred
Terraform Atlas provider version used to reproduce the issue
Terraform version used to reproduce the issue
Confirmation if Terraform OSS, Terraform Cloud, or Terraform Enterprise deployment

The ticket CLOUDP-229736 was created for internal tracking.

Hi @lsanchez-v 👋 Thanks for opening the issue! Could you also provide the piece of code in your pipeline that runs the TF provider and where you defined your vars? This would help us reproduce the issue. Out of curiosity, have you tried to run the same TF conf outside of the pipeline? did you get the same errors? Thanks!

Hi @andreaangiolillo

Yes sure, here is the part of the pipeline that runs that TF code:

.update_terraform_version: &update_terraform_version
  - git clone --depth=1 https://github.com/tfutils/tfenv.git tfenv
  - tfenv/bin/tfenv install 1.6.6
  - tfenv/bin/tfenv use 1.6.6

.export_mongodb_atlas_credentials: &export_mongodb_atlas_credentials
  - export MONGODB_ATLAS_PUBLIC_KEY=$MONGODB_ATLAS_LIVE_PUBLIC_KEY
  - export MONGODB_ATLAS_PRIVATE_KEY=$MONGODB_ATLAS_LIVE_PRIVATE_KEY

.validate_ci_script: &validate_ci_script
  - cd live_account/mongodb_atlas/per_env
  - rm -rf .terraform
  - terraform init -input=false -backend-config=environments/ci/terraform.backend -upgrade
  - terraform validate

.plan_ci_script: &plan_ci_script
  - terraform plan -input=false -var-file=environments/ci/terraform.tfvars -out $PLAN
  - terraform show --json $PLAN | jq -r $REPORT_FILTER > $PLAN_JSON
  - cat $PLAN_JSON

plan_ci:
  stage: plan
  only:
    changes:
      - live_account/mongodb_atlas/per_env/**/*
    refs:
      - merge_requests
  except:
    - master
  tags:
    - dev
  script:
    - *update_terraform_version
    - *validate_ci_script
    - *export_mongodb_atlas_credentials
    - *plan_ci_script

The variables are defined in a .tfvars file.

Indeed running the TF code from local works, it only fails from the pipeline.

Thanks!

@lsanchez-v thanks for providing the script. If running TF code from local works, it might be worth it to understand if the pipeline is adding/removing some value compared to when you are directly calling the script. Also: can you confirm that you are actually using the provider version 1.15.1 when using the pipeline?

@marcosuma Here is the output of the pipeline when installing the TF providers:

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Finding mongodb/mongodbatlas versions matching "~> 1.15"...
- Installing hashicorp/aws v5.36.0...
- Installed hashicorp/aws v5.36.0 (self-signed, key ID 34365D9472D7468F)
- Installing mongodb/mongodbatlas v1.15.1...
- Installed mongodb/mongodbatlas v1.15.1 (signed by a HashiCorp partner, key ID 2A32ED1F3AD25ABF)

Another curious thing is that we are deploying another Atlas module using the same provider version from that same pipeline, and that one works correctly, so it seems to me that it could be maybe related to the resources being deployed. Here is the TF code of the other module:

locals {
  mongodbatlas_privatelink_endpoint_attrs_to_export = ["endpoint_service_name", "project_id", "private_link_id"]
}

resource "mongodbatlas_network_container" "mongo_atlas" {
  project_id       = var.mongodb_atlas_project_id
  atlas_cidr_block = var.mongodb_atlas_cidr_block
  provider_name    = "AWS"
  region_name      = var.provider_region_name
}

#
# Private link
#
resource "mongodbatlas_privatelink_endpoint" "mongo_pl" {
  project_id    = var.mongodb_atlas_project_id
  provider_name = "AWS"
  region        = var.aws_region_name
  depends_on    = [mongodbatlas_network_container.mongo_atlas]
}

resource "aws_ssm_parameter" "privatelink_endpoint_data" {
  for_each    = {
    for k, v in mongodbatlas_privatelink_endpoint.mongo_pl : k => v
    if contains(local.mongodbatlas_privatelink_endpoint_attrs_to_export, k)
  }
  name        = "/mongodb_atlas/private_link/${each.key}"
  description = "MongoDb Atlas Private Link Endpoint Attr ${each.key}"
  type        = "String"
  value       = each.value
}

The pipeline structure for this module is exactly the same as the other one.

Indeed running the TF code from local works, it only fails from the pipeline.

Given this resolving issue, suggest escalating this item to GitLab repo. Feel free to open new issue in case you need else related to Terraform MongoDB Atlas Provider

mongodb / terraform-provider-mongodbatlas