toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.11k stars 174 forks source link

Figuring out when sync has completed #377

Open holoiii opened 1 year ago

holoiii commented 1 year ago

Is there a way to determine when sync has completed? I have a use case where:

  1. The user can update 100k+ records in postgres
  2. pgsync can take a while to update the ES index of those records
  3. I'd like to know when pgsync is finished updating the ES indices of the records

Some ideas:

  1. Check the redis queue to see if it is empty - but this seems unreliable
  2. Somehow check each record to see if it is indexed properly in ES - not sure how to do this because the schema is defined in pgsync's schema file and would need to be translated into an ES query with all the relevant fields
toluaina commented 1 year ago