search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.97k stars 4.1k forks source link

[destination-snowflake] In full replace sync, final tables are replaced with empty temp tables on source error #33230

Closed cwegrzyn closed 3 months ago

cwegrzyn commented 10 months ago

Connector Name

destination-snowflake

Connector Version

3.4.10

What step the error happened?

During the sync

Relevant information

Preface: I'm not sure whether to classify this as a source-postgres, a destination-snowflake error, or something else. The immediate undesirable consequence is in snowflake, but it's possible that the source connector or something else is what is badly behaved.

In the past month, we have had several instances of destination Snowflake tables being completely emptied during a full-refresh sync (from a postgres source) that reports a failure. The expected behavior would be that the tables just remain as they were prior to the source failure.

The causes of the source failures seem to have varied. We've seen OOM errors and terminating connection due to conflict with recovery that we think both have triggered this behavior.

In any case, what seems to happen is that after that error, the source stops fetching new data, and the destination seems to assume that all data that should be loaded has been fetched. It proceeds to load the tables that had already been fetched into temp tables. The temp tables for tables that had not been fetched remain empty. It then replaces all destination tables with their temp counterparts, whether empty or not.

In the example below, the error happens before the source reads from the users table, but as the logs show, the connector still replaces that table. I've included the source error as well as all times the destination logs mention the users table as examples.

Relevant log output

Source error:
2023-12-07 14:01:50 source > 2023-12-07 14:01:50 WARN c.z.h.p.ProxyConnection(checkException):177 - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection@1f443fae marked as broken because of SQLSTATE(08006), ErrorCode(0)
2023-12-07 14:01:50 source > org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2603) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:2235) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java) [HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:100) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) [?:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:46) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:72) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) [guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) [?:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:233) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:272) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:170) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:710) [io.airbyte.airbyte-integrations.connectors-source-postgres-0.50.34.jar:?]
2023-12-07 14:01:50 source > Caused by: java.io.EOFException
2023-12-07 14:01:50 source >    at org.postgresql.core.PGStream.receiveChar(PGStream.java:467) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2598) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    ... 33 more
2023-12-07 14:01:50 source > 2023-12-07 14:01:50 ERROR i.a.c.d.j.StreamingJdbcDatabase$1(tryAdvance):109 - SQLState: 40001, Message: FATAL: terminating connection due to conflict with recovery
2023-12-07 14:01:50 source >   Detail: User was holding a relation lock for too long.
2023-12-07 14:01:50 source >   Hint: In a moment you should be able to reconnect to the database and repeat your command.
2023-12-07 14:01:50 source > 2023-12-07 14:01:50 ERROR i.a.c.u.CompositeIterator(close):123 - exception while closing
2023-12-07 14:01:50 source > java.lang.RuntimeException: java.sql.SQLException: Connection is closed
2023-12-07 14:01:50 source >    at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:80) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323) ~[?:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.LazyAutoCloseableIterator.close(LazyAutoCloseableIterator.java:56) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.CompositeIterator.close(CompositeIterator.java:121) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) [airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:271) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:170) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) [airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:710) [io.airbyte.airbyte-integrations.connectors-source-postgres-0.50.34.jar:?]
2023-12-07 14:01:50 source > Caused by: java.sql.SQLException: Connection is closed
2023-12-07 14:01:50 source >    at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:502) ~[HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >    at jdk.proxy3/jdk.proxy3.$Proxy39.getAutoCommit(Unknown Source) ~[?:?]
2023-12-07 14:01:50 source >    at com.zaxxer.hikari.pool.HikariProxyConnection.getAutoCommit(HikariProxyConnection.java) ~[HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:72) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    ... 19 more
2023-12-07 14:01:50 source > 2023-12-07 14:01:50 ERROR i.a.c.i.b.AirbyteExceptionHandler(uncaughtException):64 - Something went wrong in the connector. See the logs for more details.
2023-12-07 14:01:50 source > java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery
2023-12-07 14:01:50 source >   Detail: User was holding a relation lock for too long.
2023-12-07 14:01:50 source >   Hint: In a moment you should be able to reconnect to the database and repeat your command.
2023-12-07 14:01:50 source >    at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:112) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) ~[?:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:46) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:72) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2023-12-07 14:01:50 source >    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:233) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:272) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:170) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:710) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.50.34.jar:?]
2023-12-07 14:01:50 source >    Suppressed: java.lang.RuntimeException: java.sql.SQLException: Connection is closed
2023-12-07 14:01:50 source >        at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:80) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323) ~[?:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.LazyAutoCloseableIterator.close(LazyAutoCloseableIterator.java:56) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.CompositeIterator.close(CompositeIterator.java:121) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53) ~[airbyte-cdk-airbyte-commons-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:271) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:170) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:710) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.50.34.jar:?]
2023-12-07 14:01:50 source >    Caused by: java.sql.SQLException: Connection is closed
2023-12-07 14:01:50 source >        at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:502) ~[HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >        at jdk.proxy3/jdk.proxy3.$Proxy39.getAutoCommit(Unknown Source) ~[?:?]
2023-12-07 14:01:50 source >        at com.zaxxer.hikari.pool.HikariProxyConnection.getAutoCommit(HikariProxyConnection.java) ~[HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >        at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:72) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >        ... 19 more
2023-12-07 14:01:50 source > Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery
2023-12-07 14:01:50 source >   Detail: User was holding a relation lock for too long.
2023-12-07 14:01:50 source >   Hint: In a moment you should be able to reconnect to the database and repeat your command.
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2598) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:2235) ~[postgresql-42.6.0.jar:42.6.0]
2023-12-07 14:01:50 source >    at com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java) ~[HikariCP-5.0.1.jar:?]
2023-12-07 14:01:50 source >    at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:100) ~[airbyte-cdk-core-0.4.7.jar:?]
2023-12-07 14:01:50 source >    ... 30 more
2023-12-07 14:01:50 INFO i.a.w.g.BufferedReplicationWorker(readFromSource):372 - readFromSource: source exception
io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1
    at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:369) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:243) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-12-07 14:01:50 INFO i.a.w.g.BufferedReplicationWorker(readFromSource):382 - readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2023-12-07 14:01:50 INFO i.a.w.g.BufferedReplicationWorker(processMessage):422 - processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2023-12-07 14:01:50 INFO i.a.w.g.BufferedReplicationWorker(writeToDestination):454 - writeToDestination: done. (forDest.isDone:true, isDestRunning:true)

...

2023-12-07 14:02:10 destination > 2023-12-07 14:02:10 INFO i.a.i.b.d.t.DefaultTyperDeduper(lambda$typeAndDedupeTask$2):216 - Waiting for raw table writes to pause for web_yr4_prod.users
2023-12-07 14:02:10 destination > 2023-12-07 14:02:10 INFO i.a.i.b.d.t.TypeAndDedupeTransaction(executeTypeAndDedupe):36 - Attempting typing and deduping for web_yr4_prod.users with suffix
2023-12-07 14:02:10 destination > 2023-12-07 14:02:10 INFO i.a.i.d.s.t.SnowflakeDestinationHandler(execute):134 - Executing sql 2bf893f8-e8d2-4e1d-9745-fda437a87886: BEGIN TRANSACTION;

...

2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.i.b.d.t.DefaultTyperDeduper(lambda$typeAndDedupeTask$2):224 - Allowing other threads to proceed for web_yr4_prod.users
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream attribution_orgs. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_attribution_orgs"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream site_options. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_site_options"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream attribution_methods. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_attribution_methods"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream queued_emails. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_queued_emails"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream ar_internal_metadata. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_ar_internal_metadata"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream active_storage_blobs. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_active_storage_blobs"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream data_deletion_requests. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_data_deletion_requests"
2023-12-07 14:02:13 destination > 2023-12-07 14:02:13 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream peer_review_questions. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_peer_review_questions"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream rejection_reasons. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_rejection_reasons"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream essay_question_answers. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_essay_question_answers"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream scholarship_applications. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_scholarship_applications"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream quiz_question_answers. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_quiz_question_answers"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream quiz_questions. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_quiz_questions"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream email_templates. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_email_templates"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream leads. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_leads"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream essay_questions. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_essay_questions"
2023-12-07 14:02:14 destination > 2023-12-07 14:02:14 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream snowflake_sync_todos. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_snowflake_sync_todos"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream data_deletion_consent_logs. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_data_deletion_consent_logs"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream rejection_reason_categories. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_rejection_reason_categories"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream active_storage_attachments. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_active_storage_attachments"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream active_storage_variant_records. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_active_storage_variant_records"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream schema_migrations. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_schema_migrations"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream essay_question_peer_reviews. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_essay_question_peer_reviews"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream sessions. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_sessions"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream users. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_users"
2023-12-07 14:02:15 destination > 2023-12-07 14:02:15 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):127 - Cleaning stage in destination started for stream project_peer_reviews. schema airbyte_internal, stage: "airbyte_internal"."web_yr4_prod_raw__stream_project_peer_reviews"
2023-12-07 14:02:16 destination > 2023-12-07 14:02:16 INFO i.a.i.b.d.t.DefaultTyperDeduper(commitFinalTables):258 - Committing final tables

...

2023-12-07 14:02:17 destination > 2023-12-07 14:02:17 INFO i.a.i.b.d.t.DefaultTyperDeduper(lambda$commitFinalTableTask$6):281 - Overwriting final table with tmp table for stream web_yr4_prod.users
2023-12-07 14:02:17 destination > 2023-12-07 14:02:17 INFO i.a.i.d.s.t.SnowflakeDestinationHandler(execute):134 - Executing sql 368dc45e-8a08-4cc4-9fc7-83afc0d79bed: BEGIN TRANSACTION;
2023-12-07 14:02:17 destination > DROP TABLE IF EXISTS "WEB_YR4_PROD"."USERS";
2023-12-07 14:02:17 destination > ALTER TABLE "WEB_YR4_PROD"."USERS_AIRBYTE_TMP" RENAME TO "WEB_YR4_PROD"."USERS";
2023-12-07 14:02:17 destination > COMMIT;
2023-12-07 14:02:17 destination > 

...

2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.i.d.s.t.SnowflakeDestinationHandler(execute):153 - Sql 368dc45e-8a08-4cc4-9fc7-83afc0d79bed completed in 639 ms

...

2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.i.b.d.t.DefaultTyperDeduper(cleanup):299 - Cleaning Up type-and-dedupe thread pool
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onCloseFunction$1):134 - Cleaning up destination completed.
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.c.i.d.AsyncStreamConsumer(close):211 - class io.airbyte.cdk.integrations.destination_async.AsyncStreamConsumer closed
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 WARN i.a.c.i.b.IntegrationRunner(stopOrphanedThreads):362 - The main thread is exiting while children non-daemon threads from a connector are still active.
2023-12-07 14:02:18 destination > Ideally, this situation should not happen...
2023-12-07 14:02:18 destination > Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
2023-12-07 14:02:18 destination > The main thread is: main (RUNNABLE)
2023-12-07 14:02:18 destination >  Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
2023-12-07 14:02:18 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.dumpThread(IntegrationRunner.java:392)
2023-12-07 14:02:18 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.stopOrphanedThreads(IntegrationRunner.java:366)
2023-12-07 14:02:18 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:189)
2023-12-07 14:02:18 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
2023-12-07 14:02:18 destination >         at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:101)
2023-12-07 14:02:18 destination >         at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationRunner.main(SnowflakeDestinationRunner.java:20)
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 WARN i.a.c.i.b.IntegrationRunner(stopOrphanedThreads):374 - Active non-daemon thread: pool-12-thread-10 (RUNNABLE)
2023-12-07 14:02:18 destination >  Thread stacktrace: java.base@17.0.8.1/sun.nio.ch.Net.poll(Native Method)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:186)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:290)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/java.net.Socket$SocketInputStream.read(Socket.java:966)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.RestRequest.execute(RestRequest.java:222)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.RestRequest.execute(RestRequest.java:67)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.core.HttpUtil.executeRequestInternal(HttpUtil.java:751)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.core.HttpUtil.executeRequest(HttpUtil.java:684)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.core.HttpUtil.executeGeneralRequest(HttpUtil.java:601)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.sendBatch(TelemetryClient.java:352)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.lambda$sendBatchAsync$0(TelemetryClient.java:282)
2023-12-07 14:02:18 destination >         at app//net.snowflake.client.jdbc.telemetry.TelemetryClient$$Lambda$309/0x00007f627c4372f8.call(Unknown Source)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07 14:02:18 destination >         at java.base@17.0.8.1/java.lang.Thread.run(Thread.java:833)
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.c.i.b.IntegrationRunner(runInternal):228 - Completed integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2023-12-07 14:02:18 destination > 2023-12-07 14:02:18 INFO i.a.c.i.b.a.AdaptiveDestinationRunner$Runner(run):102 - Completed destination: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2023-12-07 14:02:20 INFO i.a.w.g.BufferedReplicationWorker(readFromDestination):490 - readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
2023-12-07 14:02:20 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):244 - sync summary: {
  "status" : "failed",
  "recordsSynced" : 0,
  "bytesSynced" : 0,
  "startTime" : 1701957626858,
  "endTime" : 1701957740150,
  "totalStats" : {
    "bytesCommitted" : 0,
    "bytesEmitted" : 66830547,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 1701957740147,
    "destinationWriteStartTime" : 1701957626859,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 8795,
    "recordsCommitted" : 0,
    "replicationEndTime" : 1701957740149,
    "replicationStartTime" : 1701957626858,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1701957626859,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ {
...
  }, {
    "streamName" : "attribution_orgs",
    "streamNamespace" : "public",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 13975,
      "recordsEmitted" : 196,
      "recordsCommitted" : 0
    }
  }, {
    "streamName" : "users",
    "streamNamespace" : "public",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0
    }
  }, {
...
2023-12-07 14:02:20 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):245 - failures: [ {
  "failureOrigin" : "source",
  "failureType" : "config_error",
  "internalMessage" : "java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery\n  Detail: User was holding a relation lock for too long.\n  Hint: In a moment you should be able to reconnect to the database and repeat your command.",
  "externalMessage" : "We're having issues syncing from a Postgres replica that is configured as a hot standby server. Please see https://go.airbyte.com/pg-hot-standby-error-message for options and workarounds",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 27429,
    "from_trace_message" : true,
    "connector_command" : "read"
  },
  "stacktrace" : "java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery\n  Detail: User was holding a relation lock for too long.\n  Hint: In a moment you should be able to reconnect to the database and repeat your command.\n\tat io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:112)\n\tat java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)\n\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:46)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)\n\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)\n\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:72)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42)\n\tat com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)\n\tat com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)\n\tat java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:233)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:272)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:170)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)\n\tat io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:710)\n\tSuppressed: java.lang.RuntimeException: java.sql.SQLException: Connection is closed\n\t\tat io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:80)\n\t\tat java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)\n\t\tat io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)\n\t\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53)\n\t\tat io.airbyte.commons.util.LazyAutoCloseableIterator.close(LazyAutoCloseableIterator.java:56)\n\t\tat io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)\n\t\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53)\n\t\tat io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)\n\t\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53)\n\t\tat io.airbyte.commons.util.CompositeIterator.close(CompositeIterator.java:121)\n\t\tat io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106)\n\t\tat io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)\n\t\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53)\n\t\tat io.airbyte.commons.util.AutoCloseableIterators.lambda$appendOnClose$0(AutoCloseableIterators.java:106)\n\t\tat io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)\n\t\tat io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:53)\n\t\tat io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:271)\n\t\t... 3 more\n\tCaused by: java.sql.SQLException: Connection is closed\n\t\tat com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:502)\n\t\tat jdk.proxy3/jdk.proxy3.$Proxy39.getAutoCommit(Unknown Source)\n\t\tat com.zaxxer.hikari.pool.HikariProxyConnection.getAutoCommit(HikariProxyConnection.java)\n\t\tat io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:72)\n\t\t... 19 more\nCaused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery\n  Detail: User was holding a relation lock for too long.\n  Hint: In a moment you should be able to reconnect to the database and repeat your command.\n\tat org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)\n\tat org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401)\n\tat org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2598)\n\tat org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:2235)\n\tat com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java)\n\tat io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:100)\n\t... 30 more\n",
  "timestamp" : 1701957710274
}, {
  "failureOrigin" : "source",
  "internalMessage" : "Source process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the source connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 27429,
    "connector_command" : "read"
  },
  "stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:369)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:243)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n",
  "timestamp" : 1701957710504
} ]
2023-12-07 14:02:20 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-12-07 14:02:20 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END REPLICATION -----
2023-12-07 14:02:20 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-12-07 14:02:21 INFO i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):56 - Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 10 seconds
2023-12-07 14:02:22 ERROR i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):54 - Failing job: 27429, reason: Connection Check Failed 1fd0b804-ee42-41b1-8878-f5e8f8986d0d

Contribute

menishak commented 10 months ago

I think this is related to this issue:

https://github.com/airbytehq/airbyte/issues/9663

evantahler commented 4 months ago

Stay tuned - this will be solved by the Airbyte 1.0 release!

evantahler commented 3 months ago

Closed by #39473 - v3.11.0