Provider produced inconsistent result after apply - Connection

[agility-adamp]

Describe the bug I am unable to successfully add a TheTradeDesk Connection. Running terraform apply .... throws a Terraform error and the error message let's me know this is a problem with the FiveTran Terraform provider.

If I am able to get the Connection to deploy, it appears that only some of the configuration I sent in has worked because I've configured the resources to be paused=false, which should mean the resource is not paused. It's always paused when I am successful in getting the resource to half deploy (I've never had a fully successful deploy of the FiveTran Connection). The Destination and Connection "connection test"s report a successful test (when I manually trigger the test), so the credentials and IAM role permissions seem to be set up correctly.

I've tried changing most of the configuration items in the Connection and the Destination, to no avail.

To Reproduce Properly configure and create a TheTradeDesk Connection using Terraform.I can't find a way to make it work successfully, so I think all you have to do to reproduce this is to attempt to create a FiveTran TheTradeDesk Connection, using Terraform.

This is my config for the Connection and its Destination and Destination Group...

resource "fivetran_group" "ingest_group" {
  name = "ingest_${var.env}"
}

resource "fivetran_destination" "ingest_dataset" {
  group_id = fivetran_group.ingest_group.id
  service = "big_query"
  time_zone_offset = "-7"
  # daylight_saving_time_enabled = "true"
  region = var.fivetran_region
  trust_certificates = "true"
  trust_fingerprints = "true"
  run_setup_tests = "true"

  config {
    bucket = google_storage_bucket.fivetran_processing_storage.name
    data_set_location = var.region
    project_id = var.project
    secret_key = base64decode(google_service_account_key.fivetran_sa_key.private_key)
  }
}

resource "fivetran_connector" "ttd" {
  service = "the_trade_desk"
  group_id = fivetran_group.ingest_group.id
  run_setup_tests = "true"

  trust_certificates = "true"
  trust_fingerprints = "true"

  destination_schema {
    name = "some_name_i_picked"
  }

  config {
    bucket = var.raw_bucket
    data_set_name = "some_name_i_picked"
    login = data.google_secret_manager_secret_version.fivetran-login-sbx.secret_data
    password = data.google_secret_manager_secret_version.fivetran-password-sbx.secret_data
    use_service_account = "true"
    use_workspace = "true"
    workspace_name = "fivetran-ttd-workspace-${var.env}"
    workspace_same_as_source = "false"
    connection_type = "Directly"
    sync_mode = "AllAccounts"
  }

  depends_on = [
    fivetran_destination.ingest_dataset
  ]
}

Expected behavior I expected the Connection to be successfully created, unpaused and running its initial synch or at least to have begun it's synch schedule in an unpaused state.

Logs & Output

terraform apply ...
...
Plan: 1 to add, 0 to change, 0 to destroy.
module.fivetran.fivetran_connector.ttd: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.workspace_name: was
│ cty.StringVal("fivetran-ttd-workspace-sbx"), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.data_set_name: was
│ cty.StringVal("the_trade_desk_sbx"), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.connection_type: was
│ cty.StringVal("Directly"), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.workspace_same_as_source: was
│ cty.False, but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.bucket: was
│ cty.StringVal("my-cool-bucket-for-fivetran"), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.use_workspace: was cty.True, but
│ now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .config.use_service_account: was
│ cty.True, but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.fivetran.fivetran_connector.ttd, provider
│ "module.fivetran.provider[\"registry.terraform.io/fivetran/fivetran\"]"
│ produced an unexpected new value: .destination_schema.prefix: was
│ cty.StringVal("fivetran-ttd-sbx"), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
ERRO[0010] terraform invocation failed in /my/path
ERRO[0010] 1 error occurred:
    * [/my/path] exit status 1```

**Plugin version:**
1.1.21

**Additional context**
Most resources seem to successfully deploy but I cannot get the Connection deploy to be successful, from a Terraform point of view.

Also, I'm not sure if it's related but I'm not able to set the "Historical Sync Time Frame" from Terraform either.  It's possible for me to manually update this setting from the UI but I can't seem to affect change on this setting via Terraform.  I'm not sure if that's because of the bug I'm writing about or something unrelated.

[agility-adamp]

I was able to get the Connector to successfully deploy by fixing a few config items I mistakenly added to the Connector. These are the fields I removed from the Connector resource in Terraform, which I think were the reason Terraform borked while it was trying to deploy:

• bucket
• data_set_name
• use_service_account
• use_workspace
• workspace_name
• workspace_same_as_source
• connection_type
• sync_mode

So it seems that instead of reporting the errors I'd made, the FiveTran Terraform Provider errored out with the errors I pasted above.

On a side note, I was able to fix the issue about not being able to sync from ALL_TIME; I had to add the timeframe_months=ALL_TIME setting to the config block, on the Connector.

There still might be a problem and I'm a bit unsure if it's related but after I fixed these issues, I still get a Connector in the FiveTran UI that appears to need me to manually "Edit the Connection" and then click the "Save and Test" button.

This is what the revised connector config looks like:

resource "fivetran_connector" "ttd" {
  service = "the_trade_desk"
  group_id = fivetran_group.ingest_group.id
  run_setup_tests = "true"

  trust_certificates = "true"
  trust_fingerprints = "true"

  destination_schema {
    name = "the_trade_desk_${var.env}"
  }

  config {
    login = data.google_secret_manager_secret_version.fivetran-login-sbx.secret_data
    password = data.google_secret_manager_secret_version.fivetran-password-sbx.secret_data
    timeframe_months = "ALL_TIME"
    sync_mode = "AllAccounts"
  }

  depends_on = [
    fivetran_destination.ingest_dataset
  ]
}

[beevital]

There still might be a problem and I'm a bit unsure if it's related but after I fixed these issues, I still get a Connector in the FiveTran UI that appears to need me to manually "Edit the Connection" and then click the "Save and Test" button.

Hi @agility-adamp! Thank you for the detailed feedback.

Could you please share your group_id value so we could track logs for tests to investigate why connection wasn't fully setup after terraform apply.

As for "extra" fields in connector config block - it's a known issue. We are thinking about additional pre-validation step to prevent users from defining not-expected values in config block.

[beevital]

Changing the tag to documentation as if we had a documentation with example config for each connector - it would be easier to configure. We could provide additional layer of validation inside provider to prevent such situations also.

[agility-adamp]

Sounds great and thanks for the feedback. Sorry about the extra non-bug ticket and email noise; I actually thought I was looking at a bug when I filed this because it took me a while to figure out that I read the docs wrong and threw in a bunch of extra fields. Thanks again!

[agility-adamp]

@beevital I believe this is my group_id -> infiltration_bald