search

airbytehq / terraform-provider-airbyte

Terraform Provider for Airbyte API
https://api.airbyte.com/
MIT License
39 stars 16 forks source link

Terraform provider bug - unexpected response from API. Got an unexpected response code 400 - Primary key for stream: Account is already pre-defined #89

Open tdebroc opened 3 months ago

tdebroc commented 3 months ago

Dear Airbyte team

unexpected response from API. Got an unexpected response code 40 I created a connection in TF like this:

resource "airbyte_connection" "my_connection" {
  data_residency                       = "eu"
  destination_id                       = airbyte_destination_bigquery.bigquery.destination_id
  name                                 = "Connection SFSC to BQ for Account table"
  namespace_definition                 = "custom_format"
  namespace_format                     = "My_namespace"
  non_breaking_schema_updates_behavior = "propagate_columns"
  source_id                            = airbyte_source_salesforce.source_salesforce_account.source_id
  status                               = "active"
  schedule = {
    schedule_type = "manual"
  }
  configurations                       = {
    streams = [
      {
        "name"         = "Account"
        "sync_mode"    = "full_refresh_append"
      }
    ]
  }
}

But when I want to update it. I got:

{"detail":"The body of the request contains an invalid connection configuration. Primary key for stream: Account is already pre-defined. Please do NOT include a primary
│ key configuration for this stream.","type":"[https://reference.airbyte.com/reference/errors","title":"bad-request","status":400](https://reference.airbyte.com/reference/errors%22,%22title%22:%22bad-request%22,%22status%22:400)}

As you can see, I never provided primaryKey. Airbyte automatically has added it when creating the source. I tried to remove it manually in the TF state, but even that don't work, because it re-fetch all the source config before re-applying. And therefore the PATCH doesn't work,

I guess there is a bug in the Terraform provider.

This prevents us from migrating to Terraform which has impact on us. We want to migrate ASAP to TF otherwise we'll accumulate legacy code.

Thanks

Best regards

octavia-squidington-iii commented 3 months ago

Zendesk ticket #5512 has been linked to this issue.

JonsSpaghetti commented 2 months ago

Can you please try upgrading to Airbyte version https://github.com/airbytehq/airbyte-platform/releases/tag/v0.57.3 and try again?

tdebroc commented 2 months ago

Hey @JonsSpaghetti , looks like it works. Thank you very much for the fix,

Best regards

Thibaut

GabrielZangCubDigital commented 2 months ago 
resource "airbyte_connection" "my_connection" {
  configurations = {
    streams = [{
      name         = "Product"
      sync_mode    = "incremental_deduped_history"
      cursor_field = ["llCode"]
    }]
  }
  name           = "Mssql to Postgres"
  source_id      = airbyte_source_mssql.my_source_mssql.source_id
  destination_id = airbyte_destination_postgres.my_destination_postgres.destination_id
}

when i update my sync_mode to "incremental_deduped_history". I still got above issue.

airbyte_connection.my_connection: Modifying... [name=Mssql to Postgres]
╷
│ Error: unexpected response from API. Got an unexpected response code 400
│
│   with airbyte_connection.my_connection,
│   on main.tf line 65, in resource "airbyte_connection" "my_connection":
│   65: resource "airbyte_connection" "my_connection" {
│
│ **Request**:
│ PATCH /v1/connections/a72d6de6-a3e3-423e-9beb-98f9bb9a42cd HTTP/1.1
│ Host: localhost:8006
│ Accept: application/json
│ Authorization: Basic YWlyYnl0ZTpwYXNzd29yZA==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/go 0.4.1 2.262.2 1.0.0 airbyte
│
│ {"configurations":{"streams":[{"cursorField":["llCode"],"name":"Product","primaryKey":[["llCode"]],"syncMode":"incremental_deduped_history"}]},"dataResidency":"auto","name":"Mssql to
│ Postgres","namespaceDefinition":"destination","namespaceFormat":null,"nonBreakingSchemaUpdatesBehavior":"ignore","schedule":{"scheduleType":"manual"},"status":"active"}
│ **Response**:
│ HTTP/1.1 400 Bad Request
│ Content-Length: 296
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Mon, 22 Apr 2024 00:52:06 GMT
│ Server: nginx/1.25.3
│
│ {"detail":"The body of the request contains an invalid connection configuration. Primary key for stream: Product is already pre-defined. Please do NOT include a primary key configuration for this
│ stream.","type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400}
vishalprazn commented 2 months ago

Hello All

I am too getting same issue:

│ {"type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400,"detail":"The 132 │ body of the request contains an invalid connection configuration. Primary 133 │ key for stream: Account is already pre-defined. Please do NOT include a 134 │ primary key configuration for this stream."}

Difficult to progress, please advise.

Resolution (Dirty): Destroyed and Rebuild the connection.

GabrielZangCubDigital commented 2 months ago

Hello All

I am too getting same issue:

│ {"type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400,"detail":"The 132 │ body of the request contains an invalid connection configuration. Primary 133 │ key for stream: Account is already pre-defined. Please do NOT include a 134 │ primary key configuration for this stream."}

Difficult to progress, please advise.

Resolution (Dirty): Destroyed and Rebuild the connection.

  #   streams = [{
  #     name         = "Product"
  #     sync_mode    = "incremental_deduped_history"
  #     cursor_field = ["llCode"]
  #     primaryKey   = []
  #   }]

In another github issue, cannot find related link, i found the temporary solution above which works for me,

add an empty primaryKey attribute, it doesnt delete primarykey

tdebroc commented 2 months ago

Hello,

In fact the bug is not solved indeed :-( We just tried our prod migration this migration from Octavia to TF, but are completely blocked by this bug.

The workarounds can't work:

This is a huge blocker for our Terraform migration, we hope it can get fix soon,

Thanks much in advance,

tdebroc commented 2 months ago

In fact bug is fixed on the airbyte-server, not airbyte-server-api

So switching to terraform provider to this url:

server_url = "${var.AIRBYTE_URL}/api/public/v1"

fixed it :-) 🎉

tomascorrea commented 2 months ago

Hello All

I'm still geting this error with the airbyte latest version (v0.58.1) and terraform version 0.4.2. Changing the server_url didn't work for me.

Thanks much in advance,

hongbo-miao commented 3 days ago

Thanks @tdebroc !

For future people who come here, I want to point out today is actually just simply switching Terraform provider to server_url = "${var.airbyte_url}/api/public/v1". Here airbyte_url points to airbyte-webapp-svc.airbyte-namespace.

It helps