WAL push API to replace archive_command

[hlinnaka]

Currently, to stream WAL from a primary Postgres server, a standby or pg_receivewal needs to connect to the server, and request streaming replication. It would be nice to allow for a "push" API, where an extension could gather all the WAL as it's generated, and stream it to a remote server. The difference to current streaming replication is that the extension in the primary server would initiate the connection, not the other way round.

You can do that with archive_command today, but it works WAL file at a time, and it cannot be synchronous.

If we had an API for that, the built-in archive_command mechanism could be re-implemented as an extension using that API, simplifying the core server code.

[MMeent]

After overinterpreting this as 'create extensible WAL logger API replacing WAL-to-disk logging', but getting negative feedback (difficulties with supporting existing replication features), I did some more research.

We seem to have implemented our own WAL Proposer using APIs already available to shared_preload_libraries libraries:

• We register a background worker
• We register a synchronous replication slot for physical replication
• The BG worker consumes the changes from that synchronous physical replication slot

I'm not certain what extra API would be required here to specifically cater extensions that want to archive and push WAL at a record-by-record basis.

I'm fine with moving all of WALProposer into the contrib/zenith extension, so that we have less changes in the main tree w.r.t. upstream.

[hlinnaka]

Sure, if this can be implemented using the existing APIs, let's just do that.

[stepashka]

marking this as blocked to indicate suggestion from @MMeent to reconsider this

[stepashka]

this is a good patch for postgres, not urgent though