search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.73k stars 3.78k forks source link

Support for newer version of Postgres #38182

Open czomo opened 1 month ago

czomo commented 1 month ago

Topic

postgres platform 13 database

Relevant information

Currently, Airbyte requires connection to a Postgres 13 instance.

As docs states airbyte platform support only postgres13 as database backend. EOL for postgres 13 is 13 Nov 2025(Ends in 1 year and 6 months) https://endoflife.date/postgresql

Are there any plans for work on this field?

marcosmarxm commented 1 month ago

@czomo, this feature is already supported, but the documentation hasn't been updated yet. @bgroff can you confirm this?

mathetes87 commented 1 month ago

It would be great to know if newer versions are already supported. I've been trying to use neon.tech as the database and they only support versions 14 to 16. I was facing a number of errors originating from the airbyte-temporal container in a vanilla database and I assumed it was due to the postgres version!

mathetes87 commented 2 weeks ago

Please, any information on this? @bgroff

hongbo-miao commented 3 days ago

It actually supports well. I have opened the pull request and removed the limitation in the doc https://github.com/airbytehq/airbyte/pull/39510

I succeed with a clean Postgres 16.3, I have published all deployment codes using Terraform at https://github.com/hongbo-miao/hongbomiao.com/issues/17463#issuecomment-2180242019 :smiley: And all issues I met and solutions as well in first comment :wink:

The only thing is Airbyte copied temporalio/auto-setup old script instead of pointing to it, and it has not been updated to latest version yet, so current workaround solution is disabling rds.force_ssl (I posted details at https://github.com/airbytehq/airbyte/issues/39636) Anyway my deployment codes mentioned this issue and covered it as well.

Hope it helps!

mathetes87 commented 1 day ago

That is super useful to know! Thanks!