mcwhite commented 1 year ago

Terraform Core Version

1.2.9 (also not working with 1.3.7)

AWS Provider Version

4.27.0 (also not working with 4.52.0)

Affected Resource(s)

aws_iot_role_alias

Expected Behavior

Resource configuration

resource "aws_iot_role_alias" "alias" {
  alias    = var.iot-role-alias-ota
  role_arn = aws_iam_role.role-download.arn
}

should create an AWS IoT Role Alias.

Actual Behavior

Execution fails and returns error with status code 403 AccessDeniedException.

But corresponding AWS CLI command works fine.

Relevant Error/Panic Output Snippet

module.ota.aws_iot_role_alias.alias: Creating...
╷
│ Error: error creating role alias iot-ota-access for role arn:aws:iam::xxx:role/dsc-prod-ota-download-role: AccessDeniedException: 
│   status code: 403, request id: b2aa0c93-b140-4ef7-a543-79fb40b619a6
│ 
│   with module.ota.aws_iot_role_alias.alias,
│   on ota/main.tf line 382, in resource "aws_iot_role_alias" "alias":
│  382: resource "aws_iot_role_alias" "alias" {
│

Terraform Configuration Files

Any kind of configuration such as

resource "aws_iot_role_alias" "alias" {
  alias    = var.iot-role-alias-ota
  role_arn = aws_iam_role.role-download.arn
}

Tried in different cases.

Steps to Reproduce

See configuration

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

justinretzolk commented 1 year ago

Hey @mcwhite 👋 Thank you for taking the time to raise this! Can you take a look over the authentication and configuration section of the provider documentation to see if perhaps credentials are being picked up by the provider in a way you weren't anticipating?

mcwhite commented 1 year ago

Dear Justing,

Thank you for your response. I have created an AWS profile and have deposited my AWS credentials there. All actions with Terraform or directly with the AWS CLI are performed by using/refering to this profile.

So, my problem is:

I can create IAM roles, IAM, role policies, AWS resources (such as S3 buckets, etc).
I can apply those roles, policies, resources, etc using terraform apply.
I can also create an AWS IoT role alias using the AWS CLI.
And all those steps using the same profile.

But for some reason, the creation of the AWS IoT role alias - and only this - does not work with Terraform / AWS provider. In the error output, I can see that variables and ARN are set correctly - just like in the AWS CLI command, but terraform apply returns error 403 for this - and only this - resource.

I followed the example in the documentation, without any additional parameter: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iot_role_alias

And I just do not see any difference between the example, my AWS credentials and my configuration.

Is there any required additional step to enable AWS IoT role aliases with Terraform / AWS provider?

justinretzolk commented 1 year ago

Hey @mcwhite 👋 Thank you for the additional context! That definitely seems a bit odd, but ultimately the 403 error will have come from the AWS API, rather than something within Terraform. To answer your question more directly, there's nothing particular that should be required to enable Terraform to handle AWS IoT role aliases, so long as the credentials have the correct permissions.

Would it be possible to supply (redacted as needed) debug logs as well?

mcwhite commented 1 year ago

Hey @justinretzolk. This is the corresponding snippet out of the debug logs. As I can see, the session token is used at different places inside the log file without any error response. So, all other requests with same credentials seem to be fine.

2023-01-24T11:31:04.431+0100 [WARN]  Provider "registry.terraform.io/hashicorp/aws" produced an invalid plan for module.ota.aws_iot_role_alias.alias, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .credential_duration: planned value cty.NumberIntVal(3600) for a non-computed attribute
2023-01-24T11:31:04.431+0100 [INFO]  Starting apply for module.ota.aws_iot_role_alias.alias
2023-01-24T11:31:04.432+0100 [DEBUG] module.ota.aws_iot_role_alias.alias: applying the planned Create change
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Request iot/CreateRoleAlias Details:
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: POST /role-aliases/iot-ota-access HTTP/1.1
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Host: iot.eu-west-1.amazonaws.com
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.2.9 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.77 (go1.18.4; darwin; arm64)
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Length: 104
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Authorization: AWS4-HMAC-SHA256 Credential=ASIAVUWFPALJEN6NCVQM/20230124/eu-west-1/execute-api/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=476020c49586d82ab2b681f5fe3c9ec506448f61553da2a84bced403feaac283
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Type: application/json
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Date: 20230124T103104Z
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Security-Token: IQoJb3JpZ2luX2VjEIP//////////wEaCWV1LXdlc3QtMSJGMEQCIANcSkdi/n0EDTryb/jqG8tjUb8Pic4Fwf3ZRnniQiebAiBwAh59TIs+TRPuC+16dHKrrS8YJ686+nQmkLG8VzVntyq0Agjs//////////8BEAMaDDM4ODAzNDkyMTE3MCIMJAzwfs0jDlIH6RSDKogCakKXXhZpwwNKgD77BGYhsb8ccwQdycWvV4M6L6/3sQyTJuuzd2Mwykt1/0P7RXtI4NfIYShZRO7DEldb4yh2ZiQNgzoGbiiRkx4wLoetCHVyuHbu51UJcEvGOuXbSH0pi94cImrAu9wFJG8BybUtQ7kYU3WTTycvjIAAMY74ywwKAhktbKlsRwH39aKD1KPeq/M8YP+0fzDZ+0bMSZY3QIZ20T4QTV9M0Hacpwfs9LXZucNXgu6WzFzgYEiy9nE9Q8zAnIhcdlWoK4/AagT6jbE/FFhhSXIZaO0gJmyHj75J6PAfR0YNcrtSIiVhSiEOIjWcxQaE21O6e2szYvA6np2eLAbCG4k4MOfmvp4GOp4BXNpB8jdLgwiwtj5RT9jH1qrKqOKvfOtsqa/2MNAK6BrCFOj0a668vLxWDsBd6+k/z2s5w3FNXx+IQZh4IP6m4EeG0uPoLcNmzutGxVbopxisRexsHOL1OM4Pli7by9aoqK/LdQgJP59CU1V9OqcyPW8OFPuwvTdcj0p3SvwHEhTN4Hhn2zjD/+ewob3564jifL6uHSU+NRYRA6oiZwc=
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Accept-Encoding: gzip
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: {"credentialDurationSeconds":3600,"roleArn":"arn:aws:iam:: 388034921170:role/dsc-prod-ota-download-role"}
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: -----------------------------------------------------
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Response iot/CreateRoleAlias Details:
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: ---[ RESPONSE ]--------------------------------------
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: HTTP/2.0 403 Forbidden
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Length: 297
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Type: application/json
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Date: Tue, 24 Jan 2023 10:31:04 GMT
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amzn-Errortype: AccessDeniedException:http://internal.amazon.com/coral/com.amazon.coral.service/
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amzn-Requestid: e90e9d1b-5d4b-428c-8061-7f5f0d70ac60
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T11:31:04.668+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: -----------------------------------------------------
2023-01-24T11:31:04.669+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] 
2023-01-24T11:31:04.669+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Validate Response iot/CreateRoleAlias failed, attempt 0/25, error AccessDeniedException: 
2023-01-24T11:31:04.669+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5:    status code: 403, request id: e90e9d1b-5d4b-428c-8061-7f5f0d70ac60
2023-01-24T11:31:04.669+0100 [ERROR] provider.terraform-provider-aws_v4.27.0_x5: Response contains error diagnostic: tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=5c9da46e-338d-9616-6be7-e43cf5b98ed1 tf_resource_type=aws_iot_role_alias @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_severity=ERROR tf_rpc=ApplyResourceChange diagnostic_detail= diagnostic_summary="error creating role alias iot-ota-access for role arn:aws:iam::388034921170:role/dsc-prod-ota-download-role: AccessDeniedException: 
    status code: 403, request id: e90e9d1b-5d4b-428c-8061-7f5f0d70ac60" timestamp=2023-01-24T11:31:04.668+0100
2023-01-24T11:31:04.673+0100 [ERROR] vertex "module.ota.aws_iot_role_alias.alias" error: error creating role alias iot-ota-access for role arn:aws:iam::388034921170:role/dsc-prod-ota-download-role: AccessDeniedException: 
    status code: 403, request id: e90e9d1b-5d4b-428c-8061-7f5f0d70ac60

mcwhite commented 1 year ago

That's the debug output of the corresponding AWS CLI command which works fine. I also double checked the profile. So... my last idea is that the Authorization header resp. the signature gets not calculated correctly for the execute-api. But that is part of the AWS provider.

2023-01-24 14:56:09,470 - MainThread - botocore.hooks - DEBUG - Event before-call.iot.CreateRoleAlias: calling handler <function inject_api_version_header_if_needed at 0x7fa6f82e0ee0>
2023-01-24 14:56:09,470 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateRoleAlias) with params: {'url_path': '/role-aliases/iot-ota-access', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/json', 'User-Agent': 'aws-cli/2.7.24 Python/3.9.11 Darwin/22.3.0 exe/x86_64 prompt/off command/iot.create-role-alias'}, 'body': b'{"roleArn": "arn:aws:iam::388034921170:role/dsc-prod-ota-download-role", "credentialDurationSeconds": 1800}', 'url': 'https://iot.eu-west-1.amazonaws.com/role-aliases/iot-ota-access', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fa6e841ba90>, 'has_streaming_input': False, 'auth_type': None}}
2023-01-24 14:56:09,470 - MainThread - botocore.hooks - DEBUG - Event request-created.iot.CreateRoleAlias: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa6e841bb50>>

justinretzolk commented 1 year ago

Hey @mcwhite 👋 Thanks for providing that logging; I think that showed me what's wrong here. Looking at the request (specifically the last line):

2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Request iot/CreateRoleAlias Details:
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: POST /role-aliases/iot-ota-access HTTP/1.1
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Host: iot.eu-west-1.amazonaws.com
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.2.9 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.77 (go1.18.4; darwin; arm64)
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Length: 104
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Authorization: AWS4-HMAC-SHA256 Credential=ASIAVUWFPALJEN6NCVQM/20230124/eu-west-1/execute-api/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=476020c49586d82ab2b681f5fe3c9ec506448f61553da2a84bced403feaac283
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Type: application/json
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Date: 20230124T103104Z
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Security-Token: IQoJb3JpZ2luX2VjEIP//////////wEaCWV1LXdlc3QtMSJGMEQCIANcSkdi/n0EDTryb/jqG8tjUb8Pic4Fwf3ZRnniQiebAiBwAh59TIs+TRPuC+16dHKrrS8YJ686+nQmkLG8VzVntyq0Agjs//////////8BEAMaDDM4ODAzNDkyMTE3MCIMJAzwfs0jDlIH6RSDKogCakKXXhZpwwNKgD77BGYhsb8ccwQdycWvV4M6L6/3sQyTJuuzd2Mwykt1/0P7RXtI4NfIYShZRO7DEldb4yh2ZiQNgzoGbiiRkx4wLoetCHVyuHbu51UJcEvGOuXbSH0pi94cImrAu9wFJG8BybUtQ7kYU3WTTycvjIAAMY74ywwKAhktbKlsRwH39aKD1KPeq/M8YP+0fzDZ+0bMSZY3QIZ20T4QTV9M0Hacpwfs9LXZucNXgu6WzFzgYEiy9nE9Q8zAnIhcdlWoK4/AagT6jbE/FFhhSXIZaO0gJmyHj75J6PAfR0YNcrtSIiVhSiEOIjWcxQaE21O6e2szYvA6np2eLAbCG4k4MOfmvp4GOp4BXNpB8jdLgwiwtj5RT9jH1qrKqOKvfOtsqa/2MNAK6BrCFOj0a668vLxWDsBd6+k/z2s5w3FNXx+IQZh4IP6m4EeG0uPoLcNmzutGxVbopxisRexsHOL1OM4Pli7by9aoqK/LdQgJP59CU1V9OqcyPW8OFPuwvTdcj0p3SvwHEhTN4Hhn2zjD/+ewob3564jifL6uHSU+NRYRA6oiZwc=
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Accept-Encoding: gzip
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T11:31:04.432+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: {"credentialDurationSeconds":3600,"roleArn":"arn:aws:iam:: 388034921170:role/dsc-prod-ota-download-role"}

Unless this was the result of redaction, It looks like. your role_arn argument has am errant space in (arn:aws:iam:: 388034921170:role/dsc-prod-ota-download-role).

mcwhite commented 1 year ago

Oh. I think, I added the the space during manual editing. Sorry.

Here is the latest output without any modification and without the errant space.

2023-01-24T17:06:26.971+0100 [INFO]  provider.terraform-provider-aws_v4.27.0_x5: [INFO] Retrieved credentials from "AssumeRoleProvider"
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] Trying to get account information via sts:GetCallerIdentity
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go-v2] Request
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: POST / HTTP/1.1
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Host: sts.us-east-1.amazonaws.com
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.2.9 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.16.11 os/macos lang/go/1.18.4 md/GOOS/darwin md/GOARCH/arm64 api/sts/1.16.4
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Length: 43
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Amz-Sdk-Invocation-Id: 0c3cfdb5-a417-4b87-a04c-f57407c04778
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Amz-Sdk-Request: attempt=1; max=25
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Authorization: AWS4-HMAC-SHA256 Credential=ASIAVUWFPALJPKHHVOGW/20230124/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=09f3abcb21547babcf2ea1b7d1bc6551d076b04727f1c0835fd35835a77ede19
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Type: application/x-www-form-urlencoded
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Date: 20230124T160626Z
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amz-Security-Token: IQoJb3JpZ2luX2VjEIn//////////wEaCXVzLWVhc3QtMSJIMEYCIQCEDRG9oe4Tl/01qLyKWUgy+9DOZ2b9kuy4Yu4sbXGpAQIhALhPeWEt+DjK544Bvg9R2iWgUHkBhZ504smTDJerVyIiKrQCCPH//////////wEQAxoMMzg4MDM0OTIxMTcwIgwP/UTGluYxyblozNQqiALAv7KBUMP2vEOyYs4Ac3ZSURwtv3fDeCSzpuqyfUS6Q+CbZYLQ6i9AFyHBm8joo3CcPai9qFfgwy+roMxDEdw+p8PiqqcRxb7Rr1iMnzPfFpjZl9sEXMcss9rM+cXKiWBiEktR6UEsIkC0ODtcmCCAe0sCGkkffEc3zMyLRfR+CGqX/fD3Up1aqwSWCNpQMh/i2M89i+XkKzRhwt0iGmmSKTm6VJcU8GZx2xUkKW+HRDdzI7kdoXR7aaviKPyMrVq02x5Xw40D20HGlpL5go+BGmMwNIkHI+VjLiDibXA6zEcktZx/pgIy7RaDnWVsXNWb5Ic2OfbXQCSt4jngKIB8DAlC0/zlLp4wgoTAngY6nAHqPPyG5YCLwFz4Rdf9Wd3UzWkBP4axYQNyUwBNBfA2J3AIq8DcjXqXbYif15mPN84Nfqe8oBNjoR5MBRA///x0Tmv7sax72ShLwLQseQ/a2hOBr2a/y9Iv4ouSOKBM5ooGVGEfElIXXdmTQTqab5+N8ra515NvaeJNCW+Hd0/ESXdW7EIgyK0q+ONCEwmkgYG0pOoZHCO9PkC8yww=
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Accept-Encoding: gzip
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T17:06:26.971+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Action=GetCallerIdentity&Version=2011-06-15
2023-01-24T17:06:27.072+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Response iot/CreateRoleAlias Details:
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: ---[ RESPONSE ]--------------------------------------
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: HTTP/2.0 403 Forbidden
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Length: 297
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Content-Type: application/json
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: Date: Tue, 24 Jan 2023 16:06:27 GMT
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amzn-Errortype: AccessDeniedException:http://internal.amazon.com/coral/com.amazon.coral.service/
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: X-Amzn-Requestid: 1cd15090-a3a2-43d6-bb6b-6865dc179806
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: 
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: -----------------------------------------------------
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] 
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Validate Response iot/CreateRoleAlias failed, attempt 0/25, error AccessDeniedException: 
2023-01-24T17:06:27.073+0100 [DEBUG] provider.terraform-provider-aws_v4.27.0_x5:    status code: 403, request id: 1cd15090-a3a2-43d6-bb6b-6865dc179806
2023-01-24T17:06:27.073+0100 [ERROR] provider.terraform-provider-aws_v4.27.0_x5: Response contains error diagnostic: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=97f84267-0596-c7cf-34ad-155b96f9b0e5 @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto tf_proto_version=5.3 tf_resource_type=aws_iot_role_alias diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="error creating role alias iot-ota-access for role arn:aws:iam::388034921170:role/dsc-prod-ota-download-role: AccessDeniedException: 
    status code: 403, request id: 1cd15090-a3a2-43d6-bb6b-6865dc179806" tf_rpc=ApplyResourceChange timestamp=2023-01-24T17:06:27.073+0100
2023-01-24T17:06:27.075+0100 [ERROR] vertex "module.ota.aws_iot_role_alias.alias" error: error creating role alias iot-ota-access for role arn:aws:iam::388034921170:role/dsc-prod-ota-download-role: AccessDeniedException: 
    status code: 403, request id: 1cd15090-a3a2-43d6-bb6b-6865dc179806

justinretzolk commented 1 year ago

Thanks for the clarification @mcwhite 👍. So much for an easy win 🙂. I did, however, notice one other thing now that I'm looking a bit more. In the CLI debug logs, the credentialDurationSeconds argument is set to 1800, while in Terraform it's set to the default of 3600. Looking at the CLI reference guide, I noticed the following callout for credentialDurationSeconds:

This value must be less than or equal to the maximum session duration of the IAM role that the role alias references.

Can you test setting the credential_duration argument to 1800 in the Terraform configuration to match the CLI call, and see if that gets your past the 403?

mcwhite commented 1 year ago

I have tried several values for credential_duration (900, 1800, 3600, 7200), all with same result (= error 403).

In my view, ...

the credentials (access key, access secret key, session token) are correct
the payload is correct
the request except of authorization header seems to be equal to the debug log of the CLI command

So... authorization header and the calculation of the signature could still cause an error 403.

We have had a similar problem using IoT credentials for downloading objects from S3. There, we have also received error 403 when the signature was not calculated correctly.

Sounds similar to me.

mcwhite commented 1 year ago

For your interest: I have also upgraded Terraform to version 1.3.7 and AWS provider to version 4.52.0. Error remains.

justinretzolk commented 1 year ago

Thanks for the updates @mcwhite! I've marked this as a bug so that the team can take a look into this as soon as we can prioritize it.

NickDarvey commented 5 months ago

@mcwhite, you might try checking CloudTrail for the CreateRoleAlias action. In my case, my deployment role didn't have the right PassRole permissions but that was only visible in the CloudTrail event details.

justinretzolk commented 1 month ago

Hey @mcwhite 👋 I know it's been a while since you initially reported this, but were you able to check into the information that Nick provided above to see if that may apply to your scenario as well?

hashicorp / terraform-provider-aws

[Bug]: Creating aws_iot_role_alias fails with error 403 AccessDeniedException #29011

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note