search

doublecloud / transfer

Open Source Cloud Native Ingestion engine
https://double.cloud/services/doublecloud-transfer/
Apache License 2.0
31 stars 5 forks source link

Postgres to YTsaurus - is multi key operations transactional? #106

Open Arustinal opened 6 days ago

Arustinal commented 6 days ago

Is transfer guarantee that let's say 2 updates in single transaction on PostgreSQL side will be applied in one dyntable YT transaction?

laskoviymishka commented 4 days ago

The short answer to your question is that PostgreSQL transactions do map to YT transactions with some restriction. In dynamic tables, a single transaction is limited to 100,000 rows. For transactions exceeding 90,000 rows (due to internal considerations, to avoid OOM-s on tablet agent side, since agent hold hole transaction in memory), we split them into smaller chunks. However, transactions smaller than this threshold are preserved as they are from the source system. I will look for relevant tests or documentation to confirm this behavior.