When managing the datadog_logs_pipeline_order resource in terraform, there should be an option to ignore new built-in pipelines whose ID is not in the terraform state. These should be left in whichever order they are on the Datadog side and there should be no attempt to delete them, as their deletion is impossible.
Ignoring changes with
lifecycle {
ignore_changes = all
}
is not an option because then terraform will attempt to remove the built-in pipelines ID from the pipeline order, which will fail.
References
Currently, managing the datadog_logs_pipeline_order in terraform requires managing also all the pipelines in terraform, in order to have their IDs. However, Datadog automatically creates built-in pipelines
The result is constant errors like
Error: cannot map pipelines to existing ones
existing pipelines: [5oTcLTOgT3-XOr1BnugElg YYjpi_-pSUuuKO3tgSm8ZA b_zjYlmYRkKpnlpp6NLlhQ -U6aqoOSRLCkuBk6iw3kaA WCQkeGEiSxmSOCEd2QdXCA 7wrOds7eToK9_s-sFt6YXg RXPHaqShRo-SBK4M7jXNKQ fB9MzeQzRv-osqKj9iSOKw HgNFefYlR8C4f5bzmRpiYQ Edr3GtcdSMOqGXCsI7hCbg kBMS32nuTUOrqAS1XO7nmg _A5l7sy2RtyH4GW4YLMlcA 7XRMNOS8QWCZwNVK-vFZ1w -dZ4JJI9RGesf1czQul19Q]
pipeline to be updated: [5oTcLTOgT3-XOr1BnugElg YYjpi_-pSUuuKO3tgSm8ZA b_zjYlmYRkKpnlpp6NLlhQ -U6aqoOSRLCkuBk6iw3kaA WCQkeGEiSxmSOCEd2QdXCA 7wrOds7eToK9_s-sFt6YXg RXPHaqShRo-SBK4M7jXNKQ fB9MzeQzRv-osqKj9iSOKw HgNFefYlR8C4f5bzmRpiYQ Edr3GtcdSMOqGXCsI7hCbg kBMS32nuTUOrqAS1XO7nmg _A5l7sy2RtyH4GW4YLMlcA 7XRMNOS8QWCZwNVK-vFZ1w]
with module.pipeline_config.datadog_logs_pipeline_order.pipeline_order,
on ../../../modules/<redact>/main.tf line 226, in resource "datadog_logs_pipeline_order" "pipeline_order":
226: resource "datadog_logs_pipeline_order" "pipeline_order" {
When multiple people are working on the same repository/Datadog org, the terraform state can become very difficult to untangle due to interrelated dependencies. Applying one change will fail due to the pipeline order, other things will happen in the meanwhile, adding more edit conflicts, and the plan will become even more of a mess.
What resources or data sources are affected?
datadog_logs_pipeline_order datadog_logs_integration_pipeline
Feature Request
When managing the
datadog_logs_pipeline_orderresource in terraform, there should be an option to ignore new built-in pipelines whose ID is not in the terraform state. These should be left in whichever order they are on the Datadog side and there should be no attempt to delete them, as their deletion is impossible.Ignoring changes with
is not an option because then terraform will attempt to remove the built-in pipelines ID from the pipeline order, which will fail.
References
Currently, managing the
datadog_logs_pipeline_orderin terraform requires managing also all the pipelines in terraform, in order to have their IDs. However, Datadog automatically creates built-in pipelinesThe result is constant errors like
When multiple people are working on the same repository/Datadog org, the terraform state can become very difficult to untangle due to interrelated dependencies. Applying one change will fail due to the pipeline order, other things will happen in the meanwhile, adding more edit conflicts, and the plan will become even more of a mess.
Previous reports don't seem to have addressed this issue: https://github.com/DataDog/terraform-provider-datadog/issues/1495