search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.83k stars 4.06k forks source link

[destination-snowflake] Creating Snowflake destination through Terraform breaks Airbyte #34767

Open neurotroph opened 8 months ago

neurotroph commented 8 months ago

Connector Name

destination-snowflake

Connector Version

3.5.4

What step the error happened?

Configuring a new connector

Relevant information

I'm running into an issue with a Snowflake destination that I have created through Terraform on our self-hosted Airbyte environment. While creating the destination does not report any error, we run into issues whenever we want to update or interact with the destination.

Terraform code for the destination, using the official Airbyte provider (v 0.3.7):

resource "airbyte_destination_snowflake" "snowflake_core" {
  configuration = {
    credentials = {
      key_pair_authentication = {
        private_key          = "${var.snowflake_private_key}"
        private_key_password = ""
      }
    }
    database            = "DB_DEV"
    disable_type_dedupe = false
    host                = "<private>.snowflakecomputing.com"
    raw_data_schema     = "airbyte_internal"
    role                = "DEV_INGEST"
    schema              = "raw"
    username            = "tec_ingest"
    warehouse           = "DEV_WH_XS"
  }
  name          = "Snowflake DEV"
  workspace_id  = var.airbyte_workspace_id
}

When trying to use this destination in a connection, however, there is an unspecific error message:

Failure reason: The check connection failed because of an internal error Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=7f4b184f-7a47-3387-a38c-9f212827a28f, identity='1@6717a4ae360c', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

The log does not show any errors, however:

2024-02-02 08:24:45 platform > Docker volume job log path: /tmp/workspace/4/0/logs.log
2024-02-02 08:24:45 platform > Executing worker wrapper. Airbyte version: 0.50.46
2024-02-02 08:24:45 platform > Attempt 0 to save workflow id for cancellation
2024-02-02 08:24:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-02 08:24:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-02 08:24:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-02 08:24:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-02 08:24:45 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-02 08:24:45 platform > 
2024-02-02 08:24:45 platform > ----- START CHECK -----
2024-02-02 08:24:45 platform > 
2024-02-02 08:24:45 platform > Checking if airbyte/source-pipedrive:2.2.2 exists...
2024-02-02 08:24:45 platform > airbyte/source-pipedrive:2.2.2 was found locally.
2024-02-02 08:24:45 platform > Creating docker container = source-pipedrive-check-4-0-jidqy with resources io.airbyte.config.ResourceRequirements@71a611c2[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@3972c029[hosts=[api.pipedrive.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-02 08:24:45 platform > Preparing command: docker run --rm --init -i -w /data/4/0 --log-driver none --name source-pipedrive-check-4-0-jidqy --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-pipedrive:2.2.2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.46 -e WORKER_JOB_ID=4 airbyte/source-pipedrive:2.2.2 check --config source_config.json
2024-02-02 08:24:45 platform > Reading messages from protocol version 0.2.0
2024-02-02 08:24:47 platform > Check succeeded
2024-02-02 08:24:47 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@3fe51f1f[status=succeeded,message=<null>,additionalProperties={}]
2024-02-02 08:24:47 platform > 
2024-02-02 08:24:47 platform > ----- END CHECK -----
2024-02-02 08:24:47 platform > 
2024-02-02 08:24:48 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 10 seconds

The issue seems to be with some secrets in the Snowflake destination, because when trying to test, edit or delete the destination (through the Web UI), the following error message appears:

Error deleting destination: Internal Server Error: That secret was not found in the store! Coordinate: airbyte_workspace_b083f943-1f7a-41a0-a312-360d3f21dddf_secret_04f08965-5a00-4f6c-a501-a60c61957400_v1

So, the destination cannot be deleted. Making changes through Terraform also returns an error:

│ Error: unexpected response from API. Got an unexpected response code 500
│ 
│ **Request**:
│ DELETE /v1/destinations/05897713-7a4b-4cb8-9d10-893cb7b75262 HTTP/1.1
│ Host: 10.1.3.17:8006
│ Accept: */*
│ Authorization: Basic YWlyYnl0ZTpwYXNzd29yZA==
│ User-Agent: speakeasy-sdk/go 0.3.5 2.210.3 1.0.0 airbyte
│ 
│ 
│ **Response**:
│ HTTP/1.1 500 Internal Server Error
│ Content-Length: 240
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Fri, 02 Feb 2024 11:25:01 GMT
│ Server: nginx/1.25.3
│ 
│ {"type":"https://reference.airbyte.com/reference/errors","title":"unexpected-problem","status":500,"detail":"An unexpected problem has occurred. If this is an error that needs to
│ be addressed, please submit a pull request or github issue."}
╵

The only solution seems to be to complete re-setup the Airbyte installation.

Relevant log output

2024-02-02 08:24:45 platform > Docker volume job log path: /tmp/workspace/4/0/logs.log
2024-02-02 08:24:45 platform > Executing worker wrapper. Airbyte version: 0.50.46
2024-02-02 08:24:45 platform > Attempt 0 to save workflow id for cancellation
2024-02-02 08:24:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-02 08:24:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-02 08:24:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-02 08:24:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-02 08:24:45 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-02 08:24:45 platform > 
2024-02-02 08:24:45 platform > ----- START CHECK -----
2024-02-02 08:24:45 platform > 
2024-02-02 08:24:45 platform > Checking if airbyte/source-pipedrive:2.2.2 exists...
2024-02-02 08:24:45 platform > airbyte/source-pipedrive:2.2.2 was found locally.
2024-02-02 08:24:45 platform > Creating docker container = source-pipedrive-check-4-0-jidqy with resources io.airbyte.config.ResourceRequirements@71a611c2[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@3972c029[hosts=[api.pipedrive.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-02 08:24:45 platform > Preparing command: docker run --rm --init -i -w /data/4/0 --log-driver none --name source-pipedrive-check-4-0-jidqy --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-pipedrive:2.2.2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.46 -e WORKER_JOB_ID=4 airbyte/source-pipedrive:2.2.2 check --config source_config.json
2024-02-02 08:24:45 platform > Reading messages from protocol version 0.2.0
2024-02-02 08:24:47 platform > Check succeeded
2024-02-02 08:24:47 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@3fe51f1f[status=succeeded,message=<null>,additionalProperties={}]
2024-02-02 08:24:47 platform > 
2024-02-02 08:24:47 platform > ----- END CHECK -----
2024-02-02 08:24:47 platform > 
2024-02-02 08:24:48 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 10 seconds

Contribute

octavia-squidington-iii commented 2 weeks ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.