`kinesis_settings must be set when engine_name = "kinesis"` With kinesis_settings Value Set

[alordthorsen]

Description

I'm hitting

│ Error: 1 error occurred:
│       * kinesis_settings must be set when engine_name = "kinesis"
│
│
│
│   with module.replicate_oltp.aws_dms_endpoint.this["rates_kinesis_target"],
│   on .terraform/modules/replicate_oltp/main.tf line 160, in resource "aws_dms_endpoint" "this":
│  160: resource "aws_dms_endpoint" "this" {

I have re-initialized my terraform project after cleaning up the .terraform directory.

Versions

• Module version [Required]: 1.5.1

• Terraform version:

  terraform -version
  Terraform v1.1.9
  on linux_amd64

• Provider version(s):

  
  # terraform providers

Providers required by configuration: . ├── provider[registry.terraform.io/shippo/aws] ~> 3.0 ├── provider[registry.terraform.io/hashicorp/random] ├── module.dms_write_to_rating_data_sync_policy │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.35.0 ├── module.postgres_dms_instance_access │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.0.0 ├── module.replicate_oltp │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.6.0 ├── module.replication_backup │   └── provider[registry.terraform.io/shippo/aws] ~> 3.0 └── module.defaults ├── provider[registry.terraform.io/hashicorp/aws] └── provider[terraform.io/builtin/terraform]

## Reproduction Code [Required]

endpoints = { rates_kinesis_target = { endpoint_id = "${local.name}-rates-kinesis-target" endpoint_type = "target" engine_name = "kinesis"

  kinesis_settings = {
    # These options are described in
    # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html
    service_access_role_arn        = aws_iam_role.dms_write_to_rating_data_sync.arn
    stream_arn                     = data.aws_kinesis_stream.rating_replication_stream.arn
    partition_include_schema_table = true
    include_partition_value        = true
  }
}

Steps to reproduce the behavior:

Are you using workspaces?: Yes
Have you cleared the local cache (see Notice section above)?: Yes
<!-- List steps in order that led up to the issue you encountered -->

Terraform init -backend-config=envs/dev-main/backend.tf Terraform plan

## Expected behavior

Terraform should be able to find the `kinesis_settings` as described in the documentation.

## Actual behavior

`kinesis_settings` is not being discovered in the plan step.

### Terminal Output Screenshot(s)

![Screen Shot 2022-06-21 at 5 35 17 PM](https://user-images.githubusercontent.com/88306949/174918709-35ec18ef-8e45-4523-8a7d-df97e778415d.png)

## Additional context

<!-- Add any other context about the problem here -->

[alordthorsen]

Trace Logs From Terraform Plan

2022-06-22T23:50:40.484Z [WARN]  Provider "registry.terraform.io/hashicorp/aws" produced an invalid plan for module.replicate_oltp.aws_dms_endpoint.this["postgresql_source"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .tags: planned value cty.NullVal(cty.Map(cty.String)) does not match config value cty.MapValEmpty(cty.String)
      - .redshift_settings: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead
2022-06-22T23:50:40.485Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.replicate_oltp.aws_dms_endpoint.this["postgresql_source"]
2022-06-22T23:50:40.485Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.replicate_oltp.aws_dms_endpoint.this["postgresql_source"]
2022-06-22T23:50:40.485Z [ERROR] vertex "module.replicate_oltp.aws_dms_endpoint.this[\"rates_kinesis_target\"]" error: 1 error occurred:
        * kinesis_settings must be set when engine_name = "kinesis"
2022-06-22T23:50:40.485Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this[\"rates_kinesis_target\"]": visit complete, with errors
2022-06-22T23:50:40.485Z [TRACE] writeChange: recorded Create change for module.replicate_oltp.aws_dms_endpoint.this["postgresql_source"]
2022-06-22T23:50:40.486Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this[\"postgresql_source\"]": visit complete
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2022-06-22T23:50:40.485Z [TRACE] provider.terraform-provider-aws_v4.19.0_x5: Served request: @caller=github.com/hashicorp/terraform-plugin-go@v0.9.1/tfprotov5/tf5server/server.go:791 @module=sdk.proto tf_proto_version=5.2 tf_provider_addr=provider tf_resource_type=aws_dms_endpoint tf_req_id=4fa6f44f-c38e-fbf5-37af-3b50cbc67640 tf_rpc=PlanResourceChange timesta
mp=2022-06-22T23:50:40.485Z
2022-06-22T23:50:40.486Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this": dynamic subgraph encountered errors: 1 error occurred:
        * kinesis_settings must be set when engine_name = "kinesis"
2022-06-22T23:50:40.486Z [ERROR] vertex "module.replicate_oltp.aws_dms_endpoint.this" error: 1 error occurred:
        * kinesis_settings must be set when engine_name = "kinesis"
2022-06-22T23:50:40.486Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this": visit complete, with errors
2022-06-22T23:50:40.486Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this (expand)": dynamic subgraph encountered errors: 1 error occurred:
        * kinesis_settings must be set when engine_name = "kinesis"
2022-06-22T23:50:40.486Z [ERROR] vertex "module.replicate_oltp.aws_dms_endpoint.this (expand)" error: 1 error occurred:
        * kinesis_settings must be set when engine_name = "kinesis"
2022-06-22T23:50:40.486Z [TRACE] vertex "module.replicate_oltp.aws_dms_endpoint.this (expand)": visit complete, with errors
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp.output.endpoints (expand)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp.aws_dms_replication_task.this (expand)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp.output.replication_tasks (expand)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp.aws_dms_event_subscription.this (expand)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp.output.event_subscriptions (expand)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "module.replicate_oltp (close)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2022-06-22T23:50:40.486Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2022-06-22T23:50:40.486Z [INFO]  backend/local: plan operation completed
╷
│ Error: 1 error occurred:
│       * kinesis_settings must be set when engine_name = "kinesis"
│
│
│
│   with module.replicate_oltp.aws_dms_endpoint.this["rates_kinesis_target"],
│   on .terraform/modules/replicate_oltp/main.tf line 160, in resource "aws_dms_endpoint" "this":
│  160: resource "aws_dms_endpoint" "this" {

[alordthorsen]

I setup an S3 target just to test this behavior and am running into the same error with the new target

    s3_target = {
      endpoint_id   = "${local.name}-s3-target"
      endpoint_type = "target"
      engine_name   = "s3"
      ssl_mode      = "none"

      s3_settings = {
        bucket_folder             = "${var.env_name}-psql-raw-cdc"
        bucket_name               = module.replication_backup.bucket_name # to avoid https://github.com/hashicorp/terraform/issues/4149
        service_access_role_arn   = aws_iam_role.dms_write_to_s3_target.arn
        data_format               = "parquet"
        EnableStatistics = true,
        ParquetTimestampInMillisecond =  true,
        DatePartitionEnabled = false
      }
    }

same error different target

│ Error: 1 error occurred:
│       * s3_settings must be set when engine_name = "s3"
│
│
│
│   with module.replicate_oltp.aws_dms_endpoint.this["s3_target"],
│   on .terraform/modules/replicate_oltp/main.tf line 160, in resource "aws_dms_endpoint" "this":
│  160: resource "aws_dms_endpoint" "this" {

[alordthorsen]

After some further digging this settings declaration gets terraform plan to work

      kinesis_settings = {
        #  service_access_role_arn        = aws_iam_role.dms_write_to_rating_data_sync.arn
        stream_arn                     = data.aws_kinesis_stream.rating_replication_stream.arn
        partition_include_schema_table = true
        include_partition_value        = true
      }

Which lead me to https://github.com/hashicorp/terraform/issues/30937

[antonbabenko]

This issue has been resolved in version 1.5.2 :tada:

[bryantbiggs]

@alordthorsen can you try using v1.5.2 and let us know if that resolves your issue

[alordthorsen]

Ha, this was fast enough that I don't think terraform.io has updated yet. =P

│ Error: Unresolvable module version constraint
│
│ There is no available version of module "registry.terraform.io/terraform-aws-modules/dms/aws" (dms.tf:9) which matches the given version constraint. The newest available version is 1.5.1.

I'll give it a try when it's able to download.

[alordthorsen]

Cooool, this is working for me. Thank you for all the help.

Can either of you explain to me exactly why we needed to switch from try to lookup in this case? Does this indicate that try is a function to be avoided or is there some interaction with dynamic that makes try unreliable?

[bryantbiggs]

as @antonbabenko has discovered, functions such as try(), can(), jsonencode() appear to either be eagerly or exhaustively evaluated which causes issues when values within the underlying map are not yet known. lookup() does not appear to follow this same approach and instead appears to be more of a lazy evaluation - @apparentlymart might know better; I haven't gone in to the source to fully grok the difference in implementations to say definitively

The issue you linked above is well known but typically associated with for_each situations where the keys of the map that are being iterated over is not static and needs to be evaluated which leads to the common error The "for_each" value depends on resource attributes that cannot be determined until apply which is not the case here

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.