search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.1k stars 4.12k forks source link

[destination-postgres] Significant amount of Postgres log Warnings generated on syncs #31878

Closed walker-philips closed 5 months ago

walker-philips commented 1 year ago

Connector Name

destination-postgres

Connector Version

0.4.0

What step the error happened?

During the sync

Relevant information

When a sync is started, a "LOG of could not receive data from client: Connection reset by peer" followed by a significant amount of "WARNING: there is already a transaction in progress" (it seems to be the number of models being normalized + 5). Not sure if this is indicative of poor coding within the Postgres Connector.

I have out of the box logging for Postgres. Running Postgres 14.9 on Ubuntu 22.04.2 Jammy. log file is from /var/log/postrgresql.

I was debugging a Seg Fault from Postgres Crash randomly overnight ( has now happened twice) with a PID belonging to an Airbyte Sync process both times. Not sure if Airbyte is the one causing the Seg Fault or if I have a corrupted database. Either way, the logging is substantial and swelling my logs faster than any other program/regular use of Postgers.

All my existing connectors involve Postgres (either writing to or reading from).

I am reproducing the extreme logging on two separate machines each running their own instance of Airbyte.

One Airbyte version is 0.50.31 and the other is 0.44.12

Relevant log output

No response

Contribute

evantahler commented 5 months ago

We've dramatically updated this connector since this bug report was filed. Please upgrade to the latest version and try again!

walker-philips commented 5 months ago

We are upgrading Airbyte this week and will update all connectors. I will report back if we are still encountering the issue.