disqus / pgshovel

A change data capture system for PostgreSQL
Apache License 2.0
11 stars 3 forks source link

Add the ability to drop a set that has had a relay attached. #2

Open tkaemming opened 9 years ago

tkaemming commented 9 years ago

This isn't possible right now without stopping the relay process and manually issuing a query to pgq.unregister_consumer to detach the consumer.

Removing the Queue

This could be implemented as a multiple step process:

  1. In a transaction, the administration tool should add a deletion request to the queue, and drop the log trigger from the tables in the set. The administration tool should also mark the set as deleted, and no more edits should be allowed to the configuration.
  2. When a relay process encounters the deletion request it should:
    1. publish that the set has been deleted (essentially an "end of stream" marker),
    2. unsubscribe itself from the queue,
    3. attempt to drop the queue on the origin database, (discarding an error if there are still connected relays — last relay standing should be able to successfully drop the queue when it exits)
    4. exit with a successful return code.

There should also probably be an option to force unsubscribe all consumers and just drop the queue anyway (with a loud warning about possible data loss.)

Removing the Set Configuration

Since this is needed for any replication processes, this will need to be deferred until all replication processes are complete. (This also implies we'll need to be tracking replication subscriptions, similar to how PgQ operates on the origin database. This can probably be a child znode of the set configuration znode.) When a replication process encounters the end of stream marker, it should:

  1. unsubscribe itself from the consumer list (by deleting it's znode),
  2. attempt to delete the set configuration znode (discarding an error if it cannot be deleted since there are still children — the last replication process standing should be able to successfully delete the znode when it exits)
  3. exit with a successful return code.

(This is intentionally very similar to the queue removal process.)