[source-postgres] Incremental sync support removed for views

[taherajnaFA]

Connector Name

source-postgres

Connector Version

3.6.2

What step the error happened?

Configuring a new connector

Relevant information

As per this change: https://github.com/airbytehq/airbyte/pull/42108 incremental sync support for postgres views has been removed from the UI, this feature is very much needed for our usecase as stated below:

Currently, we have a use case where we want to filter the base table of our Postgres source within a certain time frame, essentially we do not wish to sync old data into our warehouse. So the way we have achieved this is by creating a view on top of the base table with the time filter applied, also we have selected the xmin column while creating the view. To Airbyte, up until now this would just behave as a raw table, all we had to do was unselect the xmin column from the schema while setting up the connection and Airbyte would sync everything incrementally to our destination.

cc: @theyueli @xiaohansong

Relevant log output

No response

Contribute

• [ ] Yes, I want to contribute

[taherajnaFA]

@theyueli @evantahler @xiaohansong Could you please take a look at this? This is blocking us from upgrading our postgres source connector version.

[evantahler]

https://github.com/airbytehq/airbyte/pull/42108 was a deliberate change to prevent data loss with the xmin sync mode. What is your use case to sync a view?

• If you are trying to avoid syncing certain sensitive columns, Airbyte offers columns selection
• Are you using your view to pre-aggregate data? If so, why is doing that in your source preferable to doing it in your destination?
• Finally, you can use a different sync mode to sync your view if really needed. (not CDC or Xmin)

[taherajnaFA]

@evantahler the only thing we are doing in the view is adding a date filter so that we don't load in historical data in our destination, everything else is identical to the source table.

The xmin sync mode is what works for us perfectly in this case.