Airbyte

[hongbo-miao]

Airbyte Deployment

Docs

• Configure the Airbyte metadata database: https://docs.airbyte.com/operator-guides/configuring-airbyte-db/#connecting-to-an-external-postgres-database

Terraform

• Set up RDS: https://developer.hashicorp.com/terraform/tutorials/aws/aws-rds

Issues

• Postgres

  • Postgres 13 is not limit any more, I succeed Postgres 16.3: https://github.com/airbytehq/airbyte/pull/39510

• airbyte-airbyte-bootloader

  • (Resolved with workaround solution)

  • Error: couldn't find key DATABASE_USER in Secret production-hm-airbyte/airbyte-airbyte-secrets

  • Related Tickets
    • https://github.com/airbytehq/airbyte/issues/22094
    • https://github.com/airbytehq/airbyte/discussions/33835
  • Solution: https://stackoverflow.com/a/78644975/2000548

• airbyte-temporal

  • (Resolved)

  • ERROR Unable to connect to SQL database. {"error": "pq: no pg_hba.conf entry for host \"172.31.45.243\", user \"airbyte_user\", database \"temporal\", no encryption", "logging-call-at": "handler.go:52"}`

  • Related Tickets
    • https://github.com/temporalio/docker-builds/issues/59
    • https://github.com/airbytehq/airbyte/issues/26089
  • (Resolved with workaround solution)

  • pq: no pg_hba.conf entry for host

  • Workaround solution:
    • Set rds.force_ssl: 0 in RDS parameter group.
  • Long-term solution:
    • My questions
    • https://stackoverflow.com/questions/78645095/pod-airbyte-temporal-failed-to-connecto-to-rds-with-rds-force-ssl-enabled
    • https://github.com/temporalio/docker-builds/issues/221
    • https://github.com/airbytehq/airbyte/issues/39636

• airbyte-worker

  • region must not be blank or empty.
  • (Resolved)

    • region must not be blank or empty.

    • Related Tickets
    • https://github.com/airbytehq/airbyte/issues/37379
    • Solution: Add this to Helm my-values.yaml

      server:
      extraEnv:
        - name: AWS_DEFAULT_REGION
          valueFrom:
            secretKeyRef:
              name: airbyte-logs-secrets
              key: LOG_S3_BUCKET_REGION

• airbyte-worker

  • (Resolved)

  • ERROR i.m.r.Micronaut(handleStartupException):343 - Error starting Micronaut server: Error instantiating bean of type [io.airbyte.workers.storage.S3StorageClient]: region must not be blank or empty.

  • Related Tickets
    • https://github.com/airbytehq/airbyte/issues/37379
    • Solution: Add this to Helm my-values.yaml

      worker:
      extraEnv:
      - name: AWS_DEFAULT_REGION
        valueFrom:
          secretKeyRef:
            name: airbyte-logs-secrets
            key: LOG_S3_BUCKET_REGION

  • Caused by: io.temporal.failure.ApplicationFailure: message='Failure executing: POST at: https://10.215.0.1:443/api/v1/namespaces/production-hm-airbyte/pods. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).', type='io.fabric8.kubernetes.client.KubernetesClientException', nonRetryable=false

  • Related Tickets
    • https://github.com/airbytehq/airbyte/issues/7211
    • Solution: Add this to Helm my-values.yaml

      global:
      serviceAccountName: airbyte-admin
      serviceAccount:
      create: true
      name: airbyte-admin

Airbyte Connectors

Terraform

Docs

• Postgres -> Snowflake: https://github.com/airbytehq/quickstarts/blob/main/postgres_snowflake_integration/infra/airbyte/main.tf
• Set up source and destination: https://reference.airbyte.com/reference/using-the-terraform-provider

Issues

• Terraform
  • (Resolved) Got "405 Not Allowed" when use Terraform
    • Related tickets
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/43
    • Solution
    • Use Terraform provider to server_url = "${var.airbyte_url}/api/public/v1", here airbyte_url points to airbyte-webapp-svc.namespace if use Kubernetes.
  • (Resolved) Got {"type":"about:blank","status":500} when use Terraform to update connection name
    • Solution
    • Manually update the connection name in the UI to match new connection name in the Terraform
  • (Resolved) "The body of the request contains an invalid connection configuration. Primary key for stream: issues is already pre-defined. Please do NOT include a primary key configuration for this stream."
    • Related tickets
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/89
    • Solution: https://github.com/airbytehq/terraform-provider-airbyte/issues/89#issuecomment-2083136163 Switch Terraform provider to server_url = "${var.airbyte_url}/api/public/v1", here airbyte_url points to airbyte-webapp-svc.namespace if use Kubernetes.
  • configurations.streams order changed when terraform apply
    • Related tickets
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/31
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/66
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/70
    • https://github.com/airbytehq/terraform-provider-airbyte/issues/83
    • Workaround solution

      lifecycle {
      ignore_changes = [
      configurations.streams
      ]
      }

    • Solution
    • Potential fix pull request: https://github.com/airbytehq/terraform-provider-airbyte/pull/111
• Airbyte worker
  • (Resolved) Unexpected error while checking connection
    Detailedl log

    ```shell 2024-07-01 06:02:02 platform > Creating pod rce-postgres-check-03c809a7-fcf4-4d21-8b77-6ae2d5598391-0-smpsw... java.net.SocketException: Socket closed at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682) at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$11(KubePodProcess.java:734) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 2024-07-01 06:02:02 platform > Unexpected: PodDefinition is null when closing in KubePodProcess. Log an error to help debugging but not blocking the process. java.net.SocketException: Socket closed at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682) at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) 2024-07-01 06:02:02 platform > Unexpected error while checking connection: io.airbyte.workers.exception.WorkerException: Failed to create pod for check step at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:147) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.63.2.jar:?] at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:227) ~[io.airbyte-airbyte-workers-0.63.2.jar:?] at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.63.2.jar:?] at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:212) ~[io.airbyte-airbyte-workers-0.63.2.jar:?] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.215.0.1:443/api/v1/namespaces/production-horizon-airbyte/pods. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) ~[kubernetes-client-api-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.utils.internal.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:314) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:98) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:1120) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:98) ~[kubernetes-client-6.12.1.jar:?] at io.airbyte.workers.process.KubePodProcess.(KubePodProcess.java:643) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193) ~[io.airbyte-airbyte-commons-worker-0.63.2.jar:?] ... 20 more Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.215.0.1:443/api/v1/namespaces/production-horizon-airbyte/pods. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589) ~[kubernetes-client-6.12.1.jar:?] at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) ~[kubernetes-client-6.12.1.jar:?] at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142) ~[kubernetes-client-api-6.12.1.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51) ~[kubernetes-client-api-6.12.1.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136) ~[kubernetes-httpclient-okhttp-6.12.1.jar:?] ... 3 more at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$10(KubePodProcess.java:716) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) ```
  • Solution: Airbyte server re-deployed, but worker not, so it cannot talk to new Airbyte server. So just need re-deploy Airbyteworker.

• Postgres source

  • ERROR: cannot read from logical replication slot "airbyte_slot". Detail: This slot has been invalidated because it exceeded the maximum reserved size.

    Detailed log

    ```shell 2024-07-01 05:37:08 source > ERROR i.a.c.i.b.s.SshWrappedSource(read):77 Exception occurred while getting the delegate read iterator, closing SSH tunnel java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: cannot read from logical replication slot "airbyte_slot" Detail: This slot has been invalidated because it exceeded the maximum reserved size. at io.airbyte.integrations.source.postgres.cdc.PostgresDebeziumStateUtil.commitLSNToPostgresDatabase(PostgresDebeziumStateUtil.java:122) ~[io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] at io.airbyte.integrations.source.postgres.cdc.PostgresCdcCtidInitializer.cdcCtidIteratorsCombined(PostgresCdcCtidInitializer.java:125) ~[io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] at io.airbyte.integrations.source.postgres.PostgresSource.getIncrementalIterators(PostgresSource.java:476) ~[io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] at io.airbyte.integrations.source.postgres.PostgresSource.getIncrementalIterators(PostgresSource.java:123) ~[io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] at io.airbyte.cdk.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.java:172) ~[airbyte-cdk-db-sources-0.20.6.jar:?] at io.airbyte.cdk.integrations.base.ssh.SshWrappedSource.read(SshWrappedSource.java:75) [airbyte-cdk-core-0.20.6.jar:?] at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275) [airbyte-cdk-core-0.20.6.jar:?] at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173) [airbyte-cdk-core-0.20.6.jar:?] at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) [airbyte-cdk-core-0.20.6.jar:?] at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:713) [io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] Caused by: org.postgresql.util.PSQLException: ERROR: cannot read from logical replication slot "airbyte_slot" Detail: This slot has been invalidated because it exceeded the maximum reserved size. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1299) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.core.v3.QueryExecutorImpl.startCopy(QueryExecutorImpl.java:968) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.core.v3.replication.V3ReplicationProtocol.initializeReplication(V3ReplicationProtocol.java:60) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.core.v3.replication.V3ReplicationProtocol.startLogical(V3ReplicationProtocol.java:44) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.replication.fluent.ReplicationStreamBuilder$1.start(ReplicationStreamBuilder.java:38) ~[postgresql-42.6.0.jar:42.6.0] at org.postgresql.replication.fluent.logical.LogicalStreamBuilder.start(LogicalStreamBuilder.java:41) ~[postgresql-42.6.0.jar:42.6.0] at io.airbyte.integrations.source.postgres.cdc.PostgresDebeziumStateUtil.commitLSNToPostgresDatabase(PostgresDebeziumStateUtil.java:113) ~[io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?] ... 9 more Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: cannot read from logical replication slot "airbyte_slot" Detail: This slot has been invalidated because it exceeded the maximum reserved size. at io.airbyte.integrations.source.postgres.cdc.PostgresDebeziumStateUtil.commitLSNToPostgresDatabase(PostgresDebeziumStateUtil.java:122) at io.airbyte.integrations.source.postgres.cdc.PostgresCdcCtidInitializer.cdcCtidIteratorsCombined(PostgresCdcCtidInitializer.java:125) at io.airbyte.integrations.source.postgres.PostgresSource.getIncrementalIterators(PostgresSource.java:476) at io.airbyte.integrations.source.postgres.PostgresSource.getIncrementalIterators(PostgresSource.java:123) at io.airbyte.cdk.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.java:172) at io.airbyte.cdk.integrations.base.ssh.SshWrappedSource.read(SshWrappedSource.java:75) at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275) at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:713) ```

  • Snowflake destination

  • No active warehouse selected in the current session. Select an active warehouse with the 'use warehouse' command.

    Detailed log

    2024-07-01 05:37:10 destination > WARN type-and-dedupe c.z.h.p.ProxyConnection(checkException):178 HikariPool-1 - Connection net.snowflake.client.jdbc.SnowflakeConnectionV1@d66a18f marked as broken because of SQLSTATE(57P03), ErrorCode(606) net.snowflake.client.jdbc.SnowflakeSQLException: No active warehouse selected in the current session.  Select an active warehouse with the 'use warehouse' command.
          at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionSub(SnowflakeUtil.java:144) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowException(SnowflakeUtil.java:77) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:501) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:407) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:482) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:199) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:133) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.SFStatement.execute(SFStatement.java:769) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:267) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:122) ~[snowflake-jdbc-3.14.1.jar:3.14.1]
          at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-5.1.0.jar:?]
          at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-5.1.0.jar:?]
          at io.airbyte.cdk.db.jdbc.DefaultJdbcDatabase.unsafeQuery(DefaultJdbcDatabase.java:129) ~[airbyte-cdk-core-0.23.2.jar:?]
          at io.airbyte.cdk.db.jdbc.JdbcDatabase.unsafeQuery(JdbcDatabase.java:200) ~[airbyte-cdk-core-0.23.2.jar:?]
          at io.airbyte.cdk.db.jdbc.JdbcDatabase.queryJsons(JdbcDatabase.java:216) ~[airbyte-cdk-core-0.23.2.jar:?]
          at io.airbyte.integrations.destination.snowflake.typing_deduping.SnowflakeV1V2Migrator.doesAirbyteInternalNamespaceExist(SnowflakeV1V2Migrator.java:42) ~[io.airbyte.airbyte-integrations.connectors-destination-snowflake.jar:?]
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.doesValidV2RawTableAlreadyExist(BaseDestinationV1V2Migrator.java:114) ~[airbyte-cdk-typing-deduping-0.23.2.jar:?]
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.shouldMigrate(BaseDestinationV1V2Migrator.java:47) ~[airbyte-cdk-typing-deduping-0.23.2.jar:?]
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.migrateIfNecessary(BaseDestinationV1V2Migrator.java:27) ~[airbyte-cdk-typing-deduping-0.23.2.jar:?]
          at io.airbyte.integrations.base.destination.typing_deduping.DefaultTyperDeduper.lambda$runMigrationsAsync$0(DefaultTyperDeduper.java:138) ~[airbyte-cdk-typing-deduping-0.23.2.jar:?]
          at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
          at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
    Stack Trace: net.snowflake.client.jdbc.SnowflakeSQLException: No active warehouse selected in the current session.  Select an active warehouse with the 'use warehouse' command.
          at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionSub(SnowflakeUtil.java:144)
          at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowException(SnowflakeUtil.java:77)
          at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:501)
          at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:407)
          at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:482)
          at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:199)
          at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:133)
          at net.snowflake.client.core.SFStatement.execute(SFStatement.java:769)
          at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677)
          at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:267)
          at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:122)
          at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
          at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
          at io.airbyte.cdk.db.jdbc.DefaultJdbcDatabase.unsafeQuery(DefaultJdbcDatabase.java:129)
          at io.airbyte.cdk.db.jdbc.JdbcDatabase.unsafeQuery(JdbcDatabase.java:200)
          at io.airbyte.cdk.db.jdbc.JdbcDatabase.queryJsons(JdbcDatabase.java:216)
          at io.airbyte.integrations.destination.snowflake.typing_deduping.SnowflakeV1V2Migrator.doesAirbyteInternalNamespaceExist(SnowflakeV1V2Migrator.java:42)
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.doesValidV2RawTableAlreadyExist(BaseDestinationV1V2Migrator.java:114)
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.shouldMigrate(BaseDestinationV1V2Migrator.java:47)
          at io.airbyte.integrations.base.destination.typing_deduping.BaseDestinationV1V2Migrator.migrateIfNecessary(BaseDestinationV1V2Migrator.java:27)
          at io.airbyte.integrations.base.destination.typing_deduping.DefaultTyperDeduper.lambda$runMigrationsAsync$0(DefaultTyperDeduper.java:138)
          at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
          at java.base/java.lang.Thread.run(Thread.java:1583)

  • Related tickets

    • https://github.com/airbytehq/airbyte/issues/9633

  • Solution

    • https://github.com/airbytehq/airbyte/issues/40615#issuecomment-2199653132

My Tickets

• https://github.com/temporalio/docker-builds/issues/221
• https://github.com/airbytehq/airbyte/issues/39636
• https://github.com/airbytehq/airbyte/issues/40715
• https://github.com/airbytehq/airbyte/issues/40690
• https://github.com/airbytehq/airbyte/issues/40220
  • https://github.com/airbytehq/terraform-provider-airbyte/pull/119
  • https://github.com/airbytehq/airbyte/pull/40716
• https://github.com/airbytehq/airbyte/pull/39518
• https://github.com/airbytehq/airbyte/pull/39510
• https://github.com/airbytehq/terraform-provider-airbyte/pull/120
• https://github.com/airbytehq/airbyte/pull/40717
• https://github.com/airbytehq/terraform-provider-airbyte/issues/121

[hongbo-miao]

Done by

• https://github.com/hongbo-miao/hongbomiao.com/pull/17475
• https://github.com/hongbo-miao/hongbomiao.com/pull/17734

[sowla]

Thanks @hongbo-miao!

(Resolved) Got {"type":"about:blank","status":500} when use Terraform to update connection name Solution Manually update the connection name in the UI to match new connection name in the Terraform

For our setup, this error was also resolved by a solution you'd suggested for another issue. Disabling airbyte-api-server, port-forwarding from the airbyte-server service and using server_url = "http://localhost:8006/api/public/v1" works for our setup (terraform provider version ~>0.4.1; Helm chart version 0.199.0 / app version 0.63.1)