Issue : Re-Create the subscription at every apply .

[ghost]

This issue was originally opened by @ap8475 as hashicorp/terraform#19593. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

terraform {
  required_version = "0.11.3"
}

Terraform Configuration Files

lambda_function.tf

resource "aws_lambda_function" "lambda_function" {
  function_name = "${var.global_environment}-${var.lambda_function_name}-${var.global_region}"
  description   = "Lambda function to process audit trail user events"

environment {
    variables = {
      environment           = "${var.global_environment}"

    }
  }

  tags {
    Build = "${var.userevents_s3_key}"
    User_Events_Topic_Arn = "${var.user_events_topic_arn}"
  }  

  depends_on = ["*******.lambda_function_log_group"]
}

resource "aws_lambda_permission" "lambda_permission_sns" {
  statement_id  = "AllowExecutionFromSNS"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.lambda_function.function_name}"
  principal     = "sns.amazonaws.com"
  source_arn    = "${aws_sns_topic.audit_trail_log_topic.arn}"
}

SNS.tf

# Subscribes to audit trail topic
resource "aws_sns_topic_subscription" "user_events_client_sqs_subscription" {
  count                  = "${local.user_event_topic_subscription_count}"
  topic_arn              = "${var.user_events_topic_arn}"
  protocol               = "sqs"
  endpoint               = "${element(split(",", var.**user_events_client_sqs_endpoint_arn**), count.index)}"
  endpoint_auto_confirms = "true"
}

terraform.tfvar

user_events_client_sqs_endpoint_arn = "arn:aws:sqs:us-east-1:XXXXX:dev-whaeuserdelete-primary-queue,arn:aws:sqs:eu-west-1:XXXXXXXXfros-test-usercontrol-queue"

Debug Output

Whenever I am creating new Lambda , its always creating new subscription , does not keep state and tries to re-create the subscription at every apply .

Crash Output

N/A

Expected Behavior

Can we prevent it to creation of new subscription if old subscription are already there . Issue : we have 20 clients and we have created subscription for them but whenever we are doing our production deployment each time client getting confirmation mail for subscription . Ho we can prevent for reconfirmation if already they confirmed .

Actual Behavior

N/A

Steps to Reproduce

N/A

Additional Context

References

N/A

[ap8475]

I think this is Terraform issue .

[jeevesdavid]

I think this refers to subscriptions to a SNS topic. If terraform is used to create SQS subscriptions to a topic, terraform tries to recreate the subscription every time an apply is done. This okay, when we do an apply for the first time, but it is strange that terraform tries to create subscriptions even for subsequent applies.

[gwkunze]

I ran into this issue as well. The aws_sns_topic_subscription resource gets flagged for recreation on every terraform plan. I checked the tfstate file and saw that the entire resource doesn't show up, this certainly explains why it tries to recreate it every time. I am however unsure as to why the created resource never gets stored in the statefile.

[gwkunze]

I dove a little bit into the code of the provider and found my error, in my case I had protocol set to "HTTPS" instead of "https". The provider does a check to see if protocol contains the string "http" (lowercase) to see if it should wait for the confirmation_timeout_in_minutes period for the subscription to finish (and there to be an ARN). Since http isn't in HTTPS it was never getting there and there was no arn to save in the state file.

Since HTTPS seems to work in the aws api it might be worth improving that contains check to handle that case instead of silently half-succeeding

[tonyxiao]

I dove a little bit into the code of the provider and found my error, in my case I had protocol set to "HTTPS" instead of "https". The provider does a check to see if protocol contains the string "http" (lowercase) to see if it should wait for the confirmation_timeout_in_minutes period for the subscription to finish (and there to be an ARN). Since http isn't in HTTPS it was never getting there and there was no arn to save in the state file.

Since HTTPS seems to work in the aws api it might be worth improving that contains check to handle that case instead of silently half-succeeding

Thank you @gwkunze ! This fixed it for me

[morsik]

I have the same issue on Terraform v0.11.14 when trying to subscribe SNS to SQS :( And yes, I'm using lowercase sqs as Protocol.

[lcasey-aarp]

This happens on latest v0.12.19 as well with sqs and lamba protocols

+ confirmation_timeout_in_minutes = 1
        endpoint                        = "arn:aws:sqs:us-east-1:XXXXXX:XXXXX"
      + endpoint_auto_confirms          = false

      + confirmation_timeout_in_minutes = 1
        endpoint                        = "arn:aws:lambda:us-east-1:XXXXX:function:XXXXX"
      + endpoint_auto_confirms          = false

[rbUUbr]

any updates?

[antonylambert]

any updates?

[JAX-SCDC]

Any updates on this? Thanks.

[piankris]

I'm also interested in the outcome of this issue because all our team members are currently swamped by a flow of ongoing subscription emails....

[thestrongdev]

Was anyone else able to resolve? I'm also running into this issue with an sns subscriptions. Every time we try to deploy, it requires us to re-subscribe to the sns topic. My protocol is already lower case "https".

[piankris]

Was anyone else able to resolve? I'm also running into this issue with an sns subscriptions. Every time we try to deploy, it requires us to re-subscribe to the sns topic. My protocol is already lower case "https".

@thestrongdev I found a workaround but it's extremely clumsy. I basically use the null_resource of terraform where I call a python script to check if the user is already subscribed. It's triggered when the list of users to subscribe to the topic changes so that a user can also be removed from subscriptions if it's removed from the list.

P.S.: we use 'email' as protocol.

resource "null_resource" "update_sns_subscriptions" {
  triggers = {
    email_list_sha = sha256(jsonencode(var.users))
  }
  depends_on = [aws_sns_topic.key_age_monitoring]

  provisioner "local-exec" {
    command = <<-EOT
      pip install boto3
      /usr/bin/python3 ${path.module}/sns_subscribe.py --users '${jsonencode(var.users)}' --topic-arn ${aws_sns_topic.key_age_monitoring.arn}
    EOT
  }
}

this triggers the script

#!/usr/bin/env python3

import argparse
import json
import os

import boto3

STAGE = os.getenv("STAGE")
AWS_REGION = os.getenv("REGION")
AWS_ACCOUNT_NUMBER = os.getenv("AWS_ACCOUNT_NUMBER")

SNS_CLIENT = boto3.client("sns", region_name=AWS_REGION)

def get_subscriptions(topic_arn: str) -> list[dict[str, str]]:
    return SNS_CLIENT.list_subscriptions_by_topic(TopicArn=topic_arn)["Subscriptions"]

def subscribe(email: str, topic_arn: str) -> None:
    filter_policy = {"email": [email]}
    SNS_CLIENT.subscribe(
        TopicArn=topic_arn,
        Protocol="email",
        Endpoint=email,
        Attributes={"FilterPolicy": json.dumps(filter_policy)},
    )

def unsubscribe(subscriptions_arn: str) -> None:
    SNS_CLIENT.unsubscribe(SubscriptionArn=subscriptions_arn)

def main() -> None:
    parser = argparse.ArgumentParser(
        description="Take user emails and the key monitoring SNS topic ARN as arguments."
    )
    parser.add_argument("--users", type=str, help="user emails")
    parser.add_argument("--topic-arn", type=str, help="SNS key monitoring topic ARN")
    args = parser.parse_args()
    users = json.loads(args.users)
    topic_arn = args.topic_arn

    subscriptions = get_subscriptions(topic_arn)
    subscription_arn_by_email: dict[str, str] = {}
    if subscriptions:
        subscription_arn_by_email = {
            subscription["Endpoint"]: subscription["SubscriptionArn"]
            for subscription in subscriptions
        }
    for user in users:
        if user not in subscription_arn_by_email.keys():
            subscribe(user, topic_arn)

    for subscribed_email in subscription_arn_by_email.keys():
        if (subscribed_email not in users) and (
            subscription_arn_by_email[subscribed_email] != "PendingConfirmation"
        ):
            subscriptions_arn = subscription_arn_by_email[subscribed_email]
            unsubscribe(subscriptions_arn)

if __name__ == "__main__":
    main()

[seanobriennn]

Was experiencing this same issue when subscribing to SQS in another AWS Account.

Example:

resource "aws_sns_topic_subscription" "subscription" {
  count                = var.create_subscription == true ? 1 : 0
  topic_arn            = aws_sns_topic.topic[0].arn
  protocol             = "sqs"
  endpoint             = "arn:aws:sqs:${local.region}:${var.cross_account_aws_account_id}:Queue-Name"
  raw_message_delivery = true
}

Was able to overcome this by polling the endpoint queue before terraform apply and hitting the SubscribeURL in browser whilst apply is ongoing. Appreciate is not a concrete solution but stopped us having the re-create subscription issue 🤞