GCS destination issue

[kuslja]

I am getting this error while trying to run terraform apply for GCS airbyte destination:

airbyte_destination_gcs.gcs_destination[0]: Creating...
Error: failure to invoke API
with airbyte_destination_gcs.gcs_destination[0],
on main.tf line 84, in resource "airbyte_destination_gcs" "gcs_destination":
84: resource "airbyte_destination_gcs" "gcs_destination" {

error serializing request body: json: error calling MarshalJSON for type
shared.DestinationGcsAuthentication: unexpected end of JSON input

Here is relevant part of my terraform code:

resource "airbyte_workspace" "workspace" {
  count = var.DEPLOY_AIRBYTE ? 1 : 0
  name  = "Test Workspace"
}

resource "airbyte_destination_gcs" "gcs_destination" {
  count = var.DEPLOY_AIRBYTE ? 1 : 0

  name         = "GCS destination"
  workspace_id = airbyte_workspace.workspace[0].workspace_id

  configuration = {
    credential = {
      destination_gcs_update_authentication_hmac_key = {
        credential_type    = "HMAC_KEY"
        hmac_key_access_id = "12334242423423...."
        hmac_key_secret    = "afdsfas..."
      }
    }
    destination_type = "gcs"
    format = {
      destination_gcs_update_output_format_parquet_columnar_storage = {
        compression_codec = "UNCOMPRESSED"
        format_type       = "Parquet"
      }
    }
    gcs_bucket_name   = "bucket-dev-raw"
    gcs_bucket_path   = "data"
    gcs_bucket_region = "eu"
  }
}

I am using airbyte provider version 0.3.4 and terraform version 1.6.3

[kev-datams]

Hello, Same problem here :/

[kuslja]

I got it working after I have changed order of attributes, by placing "name" and "workspace_id" attributes last.

[kev-datams]

Even with changed order of attributes, I still face the same error in UPDATE mode: error serializing request body: json: error calling MarshalJSON for type shared.DestinationGcsUpdateAuthentication: unexpected end of JSON input

[kuslja]

My config also stopped working, but after few changes (not using "update" versions of config keys), it started working again. Here is my current working config:

resource "airbyte_workspace" "workspace" {
  count = var.DEPLOY_AIRBYTE ? 1 : 0
  name  = "Test Workspace"
}

resource "airbyte_destination_gcs" "gcs_destination" {
  count = var.DEPLOY_AIRBYTE ? 1 : 0
  configuration = {
    credential = {
      destination_gcs_authentication_hmac_key = {
        credential_type    = "HMAC_KEY"
        hmac_key_access_id = "12334242423423...."
        hmac_key_secret    = "afdsfas..."
      }
    }
    destination_type = "gcs"
    format = {
      destination_gcs_output_format_parquet_columnar_storage = {
        compression_codec = "UNCOMPRESSED"
        format_type       = "Parquet"
      }
    }
    gcs_bucket_name   = "bucket-dev-raw"
    gcs_bucket_path   = "data"
    gcs_bucket_region = "eu"
  }
  name         = "GCS destination"
  workspace_id = airbyte_workspace.workspace[0].workspace_id
}

[kev-datams]

Still issue even with exactly same attributes order...

Do you use Airbyte OSS or Cloud ? NB: other topic but in OSS the Workspace management looks badly implemented as requires manual setup on the UI (eg: setup email address + company name) + limited to 1 workspace.

What do you mean by "(not using "update" versions of config keys)" ?

[kuslja]

Still issue even with exactly same attributes order...

Do you use Airbyte OSS or Cloud ? NB: other topic but in OSS the Workspace management looks badly implemented as requires manual setup on the UI (eg: setup email address + company name) + limited to 1 workspace.

What do you mean by "(not using "update" versions of config keys)" ?

I have Airbyte OSS, and I agree that workspace management is badly implemented. It is not even present in the GUI, so I have to manually switch between workspaces by editing url. But still I have more then 1 workspace.

By "not using "update" version of config keys, I mean that I have replaced "destination_gcs_update_authentication_hmac_key" with "destination_gcs_authentication_hmac_key". And also replaced "destination_gcs_update_output_format_parquet_columnar_storage" with "destination_gcs_output_format_parquet_columnar_storage".

I am not sure what is exactly difference, because they have same options.

[kev-datams]

Still issue even with exactly same attributes order... Do you use Airbyte OSS or Cloud ? NB: other topic but in OSS the Workspace management looks badly implemented as requires manual setup on the UI (eg: setup email address + company name) + limited to 1 workspace. What do you mean by "(not using "update" versions of config keys)" ?

I have Airbyte OSS, and I agree that workspace management is badly implemented. It is not even present in the GUI, so I have to manually switch between workspaces by editing url. But still I have more then 1 workspace.

By "not using "update" version of config keys, I mean that I have replaced "destination_gcs_update_authentication_hmac_key" with "destination_gcs_authentication_hmac_key". And also replaced "destination_gcs_update_output_format_parquet_columnar_storage" with "destination_gcs_output_format_parquet_columnar_storage".

I am not sure what is exactly difference, because they have same options.

Fine, I also use the "non update" version:

• destination_gcs_output_format_avro_apache_avro
• destination_gcs_output_format_avro_apache_avro_compression_codec_snappy
• destination_gcs_authentication_hmac_key but without success.

I suppose the issue is elsewhere... :/