Importing aws_datapipeline_pipeline_definition wants to recreate every pipeline_object

[Luminoth]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.8 on windows_amd64

• provider registry.terraform.io/hashicorp/aws v4.9.0

Affected Resource(s)

• aws_datapipeline_pipeline_definition

Expected Behavior

I've created an essentially empty aws_datapipeline_pipeline_definition:

resource "aws_datapipeline_pipeline" "dynamodb_to_s3_on_demand" {
  name = "On-Demand: DynamoDBToS3"
}

resource "aws_datapipeline_pipeline_definition" "dynamodb_to_s3_on_demand" {
  pipeline_id = aws_datapipeline_pipeline.dynamodb_to_s3_on_demand.id

    pipeline_object {
    id   = "Default"
    name = "Default"
    field {
      key          = "workerGroup"
      string_value = "workerGroup"
    }
  }
}

With that I run terraform import to import the actual config from AWS, run terraform plan to see what would get deleted, and replace my resource definition with that. Doing this, I would expect the same result I get with other resources - I've now replicated the config and so no changes should show when running terraform plan

Actual Behavior

After the above process, when I run terraform plan, what I get is essentially terraform wanting to destroy and re-create every existing pipeline object:

      - pipeline_object {
          - id   = "EmrClusterForBackup_audit" -> null
          - name = "emrc_audit" -> null

          - field {
              - key          = "coreInstanceCount" -> null
              - string_value = "1" -> null
            }
          - field {
              - key          = "coreInstanceType" -> null
              - string_value = "m3.xlarge" -> null
            }
          - field {
              - key          = "masterInstanceType" -> null
              - string_value = "m3.xlarge" -> null
            }
          - field {
              - key          = "region" -> null
              - string_value = "#{myDDBRegion}" -> null
            }
          - field {
              - key          = "releaseLabel" -> null
              - string_value = "emr-5.23.0" -> null
            }
          - field {
              - key          = "type" -> null
              - string_value = "EmrCluster" -> null
            }
        }
      + pipeline_object {
          + id   = "EmrClusterForBackup_audit"
          + name = "emrc_audit"

          + field {
              + key          = "coreInstanceCount"
              + string_value = "1"
            }
          + field {
              + key          = "coreInstanceType"
              + string_value = "m3.xlarge"
            }
          + field {
              + key          = "masterInstanceType"
              + string_value = "m3.xlarge"
            }
          + field {
              + key          = "region"
              + string_value = "#{myDDBRegion}"
            }
          + field {
              + key          = "releaseLabel"
              + string_value = "emr-5.23.0"
            }
          + field {
              + key          = "type"
              + string_value = "EmrCluster"
            }
        }

I don't know if the issue is that there's an order to the pipeline_objects that I'm missing or what, but so far I haven't been able to figure out how to accurately import this without it trying to recreate all of the pipeline_objects.

[justinretzolk]

Hey @Luminoth 👋 Thank you for taking the time to raise this! I took a look and pipeline_object and pipeline_object.field are both TypeSets, (reference here and here), so I don't believe that ordering is the issue here.

So that we have a bit more information to go off of, can you supply the final Terraform configuration after making the necessary modifications to align with the existing infrastructure, as well as debug logs (redacted as necessary)?

Since this is related to an import, if there are other resources in this configuration, it may be helpful to copy this resource definition out into a separate configuration and reproduce the issue there to cut out noise, but this isn't strictly necessary if it would cause undue burden 🙂.

[Luminoth]

Hi @justinretzolk ! I think this should hopefully be what you're looking for:

The resource config: https://gist.github.com/Luminoth/568ddecea7e13beb9886177af61535b8

Debug output from terraform plan: https://gist.github.com/Luminoth/3d2570709cf9131446fac3ee8b2091c0

I believe I redacted all of the important bits, but if you see something I've missed, let me know and I can rebuild those.

[Luminoth]

hi @justinretzolk, I just wanted to check back on this and see if those gists were helpful or if there was anything else I could provide that might be use.

[justinretzolk]

Hey @Luminoth 👋 Thanks for getting that information over! That looks like it should be enough for now. Just as a point of clarification to set expectations, I am the community manager for the AWS Provider, so my involvement here is mostly just to triage and make sure that we've got enough information so that when someone (either one of our engineers or someone from the community) goes to take a more in-depth look, that they have the information they'll need.

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.