Terraform: Unable to fetch connection strings when using 'data' resource for existing cluster

[srashish]

Terraform CLI and Terraform MongoDB Atlas Provider Version

Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/aws v3.31.0
+ provider registry.terraform.io/terraform-providers/mongodbatlas v0.8.2

Terraform Configuration File

• I have followed the aws-privatelink example here aws-privatelink the only difference is that I'm setting it on an existing cluster, so sharing the code for that

altas_cluster.tf

data "mongodbatlas_cluster" "cluster-atlas" {
project_id                   = var.atlasprojectid
name                         = "mongoatlas-cluster-xxxxxxx"
depends_on                   = [time_sleep.wait_300_seconds]
}
output "atlasclusterstring" {
     value = data.mongodbatlas_cluster.cluster-atlas.connection_strings

}
output "plstring" {
value = lookup(data.mongodbatlas_cluster.cluster- 
atlas.connection_strings[0].aws_private_link, aws_vpc_endpoint.ptfe_service.id)
}

Steps to Reproduce

1. terraform init
2. terraform apply

Expected Behavior

atlasclusterstring = [
  {
    "aws_private_link" = {
      "vpce-0ebb76559e8affc96" = "mongodb://pl-0-us-east-1.za3fb.mongodb.net:1024,pl-0-us-east-1.za3fb.mongodb.net:1025,pl-0-us-east-1.za3fb.mongodb.net:1026/?ssl=true&authSource=admin&replicaSet=atlas-d177ke-shard-0"
    }
    "aws_private_link_srv" = {
      "vpce-0ebb76559e8affc96" = "mongodb+srv://cluster-atlas-pl-0.za3fb.mongodb.net"
    }
    "private" = ""
    "private_srv" = ""
    "standard" = "mongodb://cluster-atlas-shard-00-00.za3fb.mongodb.net:27017,cluster-atlas-shard-00-01.za3fb.mongodb.net:27017,cluster-atlas-shard-00-02.za3fb.mongodb.net:27017/?ssl=true&authSource=admin&replicaSet=atlas-d177ke-shard-0"
    "standard_srv" = "mongodb+srv://cluster-atlas.za3fb.mongodb.net"
  },
]

Actual Behavior

data.mongodbatlas_cluster.cluster-atlas.connection_strings[0].aws_private_link is empty map of string

Additional Context

I'm creating privatelink on an existing cluster and according to the example I need to pass in cluster name in the lookup function to fetch the pl_string so, for that purpose, I'm using "data" resource of terraform to get cluster information and then passing it to lookup function

References

aws-atlas-privatelink

[themantissa]

Thank you for the report @srashish. Looks like perhaps the state of the existing cluster is not getting refreshed with the new stings. I'm assuming a refresh was run?

@nikhil-mongo would you be able to repro since we have no logs included here? That speeds up our ability to narrow in quickly.

[nikhil-mongo]

@srashish Please share the cluster URL for which you are trying to do this. You can share the URL at nikhil.singh@mongodb.com for privacy.

[nikhil-mongo]

@srashish Also another question -

• Did you perform terraform import to bring the existing cluster details to the terraform.tfstate file? If not, then this is the expected behaviour. Please confirm.

[srashish]

@nikhil-mongo I used 'data' keyword to fetch details of the existing cluster.

[nikhil-mongo]

Hi @srashish The data block only has 2 arguments by default, which are project_id and name. And this is why when the plan runs, it says this is empty string. In order to get the connection string, use terraform import command to import the cluster into terraform and then refer to the connection string from the resource block directly.

[srashish]

@nikhil-mongo I think the issue is misunderstood here. I'm creating privatelink resource on an existing cluster. The privatelink is successfully created but there's a delay in its status, due to that delay I get an empty map error on the output.

Using data keyword I'm reading everything of the cluster but terraform reads it way before clearing the privatelink link, so for now I'm using a workaround with a custom delay and depends on. Ideally, this delay should be handled on the provider side.

Please let me know if there's still any confusion.

[zf-tomdesmet]

We are facing a possible related issue when creating new clusters with a private VPC endpoint where the private endpoint details are not available after the cluster and endpoint reported completed by Terraform.

mongodbatlas_cluster.this.connection_strings[0].private_endpoint is empty list of object`

A re-run of the same terraform apply right after such failure gives back the correct output.

This does however not happen on every deployment we do, but in rare occasions this issue is there.

[srashish]

We are facing a possible related issue when creating new clusters with a private VPC endpoint where the private endpoint details are not available after the cluster and endpoint reported completed by Terraform.

mongodbatlas_cluster.this.connection_strings[0].private_endpoint is empty list of object`

A re-run of the same terraform apply right after such failure gives back the correct output.

This does however not happen on every deployment we do, but in rare occasions this issue is there.

@nikhil-mongo This is same what I'm facing. I'm able to resolve it with the workaround I mentioned above. I hope it helps to resolve the issue soon.

[themantissa]

@shum any thoughts here on the delay?

[mirstan]

My guess by looking at what appears in MongoDB web UI is that your provider is returning as soon as Atlas Endpoint Service Status is Ready for connection requests doesn't wait for Endpoint Status to become Available.

If you don't want to do this by default, you could add wait_for ala kubernetes provider: https://www.hashicorp.com/blog/wait-conditions-in-the-kubernetes-provider-for-hashicorp-terraform

[themantissa]

Just looked and we are waiting for available. I think what might be happening is the cluster resource is not being updated with the new connection string in the same apply - which is why the 2nd apply works. I'll check with our developers to verify my assumption here.

[themantissa]

Internal ticket INTMDB-198

[srashish]

Internal ticket INTMDB-198

@themantissa Any updates here? Or how long the fix will take?

[themantissa]

@srashish we are investigating a way to address this timing issue and hope to have an improvement for the next provider version. Unknown on ETA at this time.

[zf-tomdesmet]

Can this be put out into a new release please?

[themantissa]

@zf-tomdesmet this will be in our next release, 1.0.0. We are working to finish up the required items and hope to release in the near term. Thanks!

[themantissa]

1.0.0 is out today, thanks!

[mirstan]

Thank you, will give it a try.