search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.25k stars 4.15k forks source link

[source-bigquery] Premature EOF error while syncing BigQuery source to PostgreSQL in Airbyte #27999

Open jenisha0512 opened 1 year ago

jenisha0512 commented 1 year ago

Connector Name

source-bigquery

Connector Version

v0.3.0

What step the error happened?

During the sync

Revelant information

I am encountering a "Premature EOF" error when syncing data from a BigQuery source to a PostgreSQL destination in Airbyte. After syncing approximately 12 GBs of data, the process fails with the following error:

java.io.EOFException: Failure reason: com.google.cloud.bigquery.BigQueryException: Premature EOF

The data I am trying to sync consists of approximately 45 million rows to my local PostgreSQL instance.

Steps to Reproduce:

  1. Set up an Airbyte connection between a BigQuery source and a PostgreSQL destination.
  2. Start the data sync process.
  3. Observe the process failing after syncing around 12 GBs of data.
  4. The error message mentioned above is displayed.

Expected Behavior:

I expect the data transfer process to complete successfully without encountering any errors.

Environment Details:

Airbyte version: 0.50.6 Deployment: Docker Source Connector and version: Bigquery (v0.3.0) Destination Connector and version: Postgres (v0.4.0)

Logs:

logs.txt

Any assistance or guidance on resolving this issue would be greatly appreciated.

Please let me know if you need any additional information.

Relevant log output

2023-07-06 01:25:59 source > INFO i.a.i.s.r.AbstractDbSource(lambda$createReadIterator$5):422 Reading stream bp_table. Records read: 43540000
2023-07-06 01:25:59 INFO i.a.w.g.ReplicationWorkerHelper(processMessageFromSource):211 - Records read: 43545000 (12 GB)
2023-07-06 01:25:59 INFO i.a.w.g.ReplicationWorkerHelper(processMessageFromSource):211 - Records read: 43550000 (12 GB)
2023-07-06 01:25:59 source > INFO i.a.i.s.r.AbstractDbSource(lambda$createReadIterator$5):422 Reading stream bp_table. Records read: 43550000
2023-07-06 01:26:21 source > INFO i.a.i.s.r.AbstractDbSource(lambda$read$1):173 Closing database connection pool.
2023-07-06 01:26:21 source > INFO i.a.i.s.r.AbstractDbSource(lambda$read$1):175 Closed database connection pool.
2023-07-06 01:26:21 source > ERROR i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 Something went wrong in the connector. See the logs for more details. com.google.cloud.bigquery.BigQueryException: Premature EOF
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.listTableData(HttpBigQueryRpc.java:527) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.bigquery.BigQueryImpl$29.call(BigQueryImpl.java:1152) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.bigquery.BigQueryImpl$29.call(BigQueryImpl.java:1147) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103) ~[gax-2.23.2.jar:2.23.2]
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[google-cloud-core-2.12.0.jar:2.12.0]
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[google-cloud-core-2.12.0.jar:2.12.0]
    at com.google.cloud.bigquery.BigQueryImpl.listTableData(BigQueryImpl.java:1146) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.bigquery.BigQueryImpl.access$500(BigQueryImpl.java:60) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.bigquery.BigQueryImpl$TableDataPageFetcher.getNextPage(BigQueryImpl.java:200) ~[google-cloud-bigquery-2.23.2.jar:2.23.2]
    at com.google.cloud.PageImpl.getNextPage(PageImpl.java:116) ~[google-cloud-core-2.12.0.jar:2.12.0]
    at com.google.cloud.PageImpl$PageIterator.computeNext(PageImpl.java:66) ~[google-cloud-core-2.12.0.jar:2.12.0]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1855) ~[?:?]
    at com.google.common.collect.CollectSpliterators$1.tryAdvance(CollectSpliterators.java:122) ~[guava-31.1-jre.jar:?]
    at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292) ~[?:?]
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206) ~[?:?]
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169) ~[?:?]
    at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298) ~[?:?]
    at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) ~[?:?]
    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:46) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:72) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[io.airbyte-airbyte-commons-24.0.2.jar:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
    at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
    at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$2(IntegrationRunner.java:200) ~[io.airbyte.airbyte-integrations.bases-base-java-24.0.2.jar:?]
    at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:270) ~[io.airbyte.airbyte-integrations.bases-base-java-24.0.2.jar:?]
    at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:199) ~[io.airbyte.airbyte-integrations.bases-base-java-24.0.2.jar:?]
    at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:140) ~[io.airbyte.airbyte-integrations.bases-base-java-24.0.2.jar:?]
    at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99) ~[io.airbyte.airbyte-integrations.bases-base-java-24.0.2.jar:?]
    at io.airbyte.integrations.source.bigquery.BigQuerySource.main(BigQuerySource.java:210) ~[io.airbyte.airbyte-integrations.connectors-source-bigquery-24.0.2.jar:?]
Caused by: java.io.IOException: Premature EOF

Contribute

octavia-squidington-iii commented 2 weeks ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.