search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.87k stars 9.21k forks source link

[Bug]: aws_chatbot_teams_channel_configuration returns "Error: waiting for creation AWS Chatbot Teams Channel Configuration" during apply, but the resource is created in AWS #38943

Open yolanv opened 3 months ago

yolanv commented 3 months ago

Terraform Core Version

1.5.7

AWS Provider Version

5.61.0

Affected Resource(s)

aws_chatbot_teams_channel_configuration

Expected Behavior

The creation of the channel configuration should succeed and terraform apply should return with success and the resource should be available in AWS.

Actual Behavior

The resource is available in AWS, but terraform apply fails with the following message:

│ Error: waiting for creation AWS Chatbot Teams Channel Configuration (<teams team id obfuscated>)
│ 
│   with module.chatbot.aws_chatbot_teams_channel_configuration.chatbot_notifications,
│   on chatbot/teams.tf line 19, in resource "aws_chatbot_teams_channel_configuration" "chatbot_notifications":
│   19: resource "aws_chatbot_teams_channel_configuration" "chatbot_notifications" {
│ 
│ couldn't find resource (21 retries)

Running terraform apply a second time returns the following error:

│ Error: creating AWS Chatbot Teams Channel Configuration (<teams team id obfuscated>)
│ 
│   with module.chatbot.aws_chatbot_teams_channel_configuration.chatbot_notifications,
│   on chatbot/teams.tf line 19, in resource "aws_chatbot_teams_channel_configuration" "chatbot_notifications":
│   19: resource "aws_chatbot_teams_channel_configuration" "chatbot_notifications" {
│ 
│ operation error chatbot: CreateMicrosoftTeamsChannelConfiguration, https
│ response error StatusCode: 409, RequestID:
│ <request id>, ConflictException: Resource with arn
│ arn:aws:chatbot::account-id:chat-configuration/microsoft-teams-channel/chatbot-notifications
│ already exists.
╵

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_chatbot_teams_channel_configuration" "chatbot_notifications" {
  configuration_name = var.configuration_name
  iam_role_arn       = aws_iam_role.chatbot_notifications.arn

  channel_id = var.channel_id
  team_id    = var.team_id
  tenant_id  = var.tenant_id

  team_name     = var.team_name
  channel_name  = var.channel_name
  logging_level = "INFO"

  sns_topic_arns = [var.sns_topic_arn]
}

Steps to Reproduce

run terraform apply which should create a chatbot configuration channel

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 3 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

DanielRieske commented 3 months ago

Hi @yolanv 👋 - thank you for raising this issue!

This resource has been made on a best-effort basis because it's a bit difficult to create a working environment to test these types of resources.

The issue here is that the create action is waiting until we are able to find the created resource and this seems to be timing out because it's unable to find the created resource. In my initial assessment I don't see anything wrong with the implementation and it would be extremely helpful if you can provide some trace logging.

Would it be possible to re-run the apply with TF_LOG=debug enable, redact where applicable data and sent this as a gist? I would be very interested in the API requests and responses from the AWS Chatbot API.

Thanks in advance!

yolanv commented 3 months ago

Hello @DanielRieske,

Sorry for the late answer. I have tried to reproduce the process to generate the debug logs on a test account in our AWS environment and we found out that the chatbot creation process was successful here. So the problem is limited to the account mentioned in the first post in this bug report.

This doesn't seem to be a Terraform issue, so we will investigate further on our end and report back if we find out what the issue was. Thank you for replying and for taking the time to provide assistance to the bug report.

yolanv commented 3 months ago

Upon further investigation, it looks like there is no clear difference as to why it works on the test account and why it doesn't work on the account it is supposed to be on. Looking at the debug logs and CloudTrail, the requests that are sent and the responses that are received are identical to each other (except account-bound information such as the account ID). I do not see any differences in the request and response bodies. All information is identical to each other.

Here is an example of a response body that is shown in the debug log on the account where it works:

{
    "NextToken": null,
    "TeamChannelConfigurations": [
        {
            "ChannelId": "<teams channel id>",
            "ChannelName": null,
            "ChatConfigurationArn": "arn:aws:chatbot::<account-id>:chat-configuration/microsoft-teams-channel/chatbot-notifications",
            "ConfigurationName": "chatbot-notifications",
            "GuardrailPolicyArns": [
                "arn:aws:iam::aws:policy/AdministratorAccess"
            ],
            "IamRoleArn": "arn:aws:iam::<account-id>:role/service-role/AWSChatbot-role",
            "LoggingLevel": "INFO",
            "SnsTopicArns": [
                "arn:aws:sns:eu-west-1:<account-id>:infra-test"
            ],
            "State": null,
            "StateReason": null,
            "Tags": [<redacted>],
            "TeamId": "<teams-team-id>",
            "TeamName": null,
            "TenantId": "<teams-tenant-id>",
            "UserAuthorizationRequired": false
        }
    ]
}

And here it timeouts: 

{
    "NextToken": null,
    "TeamChannelConfigurations": [
        {
            "ChannelId": "<teams channel id>",
            "ChannelName": null,
            "ChatConfigurationArn": "arn:aws:chatbot::<account-id>:chat-configuration/microsoft-teams-channel/chatbot-notifications",
            "ConfigurationName": "chatbot-notifications",
            "GuardrailPolicyArns": [
                "arn:aws:iam::aws:policy/AdministratorAccess"
            ],
            "IamRoleArn": "arn:aws:iam::<account-id>:role/service-role/AWSChatbot-role",
            "LoggingLevel": "INFO",
            "SnsTopicArns": [
                "arn:aws:sns:eu-west-1:<account-id>:central-eventbridge-sns-topic"
            ],
            "State": null,
            "StateReason": null,
            "Tags": [<redacted>],
            "TeamId": "<teams team id>",
            "TeamName": null,
            "TenantId": "<teams tenant id>",
            "UserAuthorizationRequired": false
        }
    ]
}

We will continue to investigate but we do not see any clear information as to why it happens.

rtkmhart commented 1 month ago

I can reliably reproduce this, it occurs when creating a second teams channel configuration in an AWS account.

Creating the first channel works fine, but creating a second channel, with a different SNS topic and Teams channel, fails in the same manner as above, the provider times out and never finds it. The configuration exists, you can see it in the console, via API (python & go), and in the debug logs below, but the provider doesn't recognize it.

With TF_LOG=debug the request line is:

2024-11-01T19:37:30.437Z [DEBUG] provider.terraform-provider-aws: HTTP Request Sent:
  http.request.body=
  | {"TeamId":"048113e8-d452-4921-95dd-be5f410e7aaf"}

and the formatted and slightly redacted response is:

{
    "NextToken": null,
    "TeamChannelConfigurations": [
        {
            "ChannelId": "19%3A4c64f77fdeb84f298d610f7030b1e13e%40thread.tacv2",
            "ChannelName": null,
            "ChatConfigurationArn": "arn:aws:chatbot::123456789012:chat-configuration/microsoft-teams-channel/UnitTest-UnitTest-Teams-us-west-2",
            "ConfigurationName": "UnitTest-UnitTest-Teams-us-west-2",
            "GuardrailPolicyArns": [
                "arn:aws:iam::123456789012:policy/UnitTest-UnitTest-ChatBot-Guardrails20241101193338876900000001"
            ],
            "IamRoleArn": "arn:aws:iam::123456789012:role/UnitTest-UnitTest-ChatBot-Teams20241101193338878500000002",
            "LoggingLevel": "INFO",
            "SnsTopicArns": [
                "arn:aws:sns:us-west-2:123456789012:apres-alerting-TeamsTest"
            ],
            "State": "ENABLED",
            "StateReason": null,
            "Tags": [
                "redacted"
            ],
            "TeamId": "048113e8-d452-4921-95dd-be5f410e7aaf",
            "TeamName": null,
            "TenantId": "REDACTED",
            "UserAuthorizationRequired": false
        },
        {
            "ChannelId": "19%3Ad1ecad0ba1c94c0abe400c93ad533123%40thread.tacv2",
            "ChannelName": null,
            "ChatConfigurationArn": "arn:aws:chatbot::123456789012:chat-configuration/microsoft-teams-channel/UnitTest-UnitTest-Teams-us-east-2",
            "ConfigurationName": "UnitTest-UnitTest-Teams-us-east-2",
            "GuardrailPolicyArns": [
                "arn:aws:iam::123456789012:policy/UnitTest-UnitTest-ChatBot-Guardrails20241101192113202400000003"
            ],
            "IamRoleArn": "arn:aws:iam::123456789012:role/UnitTest-UnitTest-ChatBot-Teams20241101192113200200000001",
            "LoggingLevel": "INFO",
            "SnsTopicArns": [
                "arn:aws:sns:us-east-2:123456789012:apres-alerting-TeamsTest"
            ],
            "State": "ENABLED",
            "StateReason": null,
            "Tags": [
                "redacted"
            ],
            "TeamId": "048113e8-d452-4921-95dd-be5f410e7aaf",
            "TeamName": null,
            "TenantId": "REDACTED",
            "UserAuthorizationRequired": false
        }
    ]
}

The second channel config I tried to create which failed, is the first one in the response, with the configuration name "UnitTest-UnitTest-Teams-us-west-2".

I tried to follow the logic in the provider code and can see from https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/chatbot/teams_channel_configuration.go#L366 that it looks up the config by team, but I don't see how it looks for the specific channel.