disqus / pgshovel

A change data capture system for PostgreSQL
Apache License 2.0
11 stars 3 forks source link

Allow batching writes with publisher. #4

Open tkaemming opened 9 years ago

tkaemming commented 9 years ago

Right now, the publisher only publishes a single mutation per publish operation. This is very slow, since it's intentionally synchronous and requires a round trip to the stream backend as well as waiting for any required acknowledgements. This should be able to batch publish operations more intelligently by queueing them until a commit occurs, or some other condition such as a maximum number of outstanding messages, crossing a byte size threshold, time between publish threshold, etc.

This also assumes that the stream backend is applying multiple messages as part of the same publish request atomically, or at least in order. This needs to be documented.