search

medic / cht-sync

Data synchronization between CouchDB and PostgreSQL for the purpose of analytics.
GNU General Public License v3.0
4 stars 5 forks source link

Use http output with json batch #102

Closed njuguna-n closed 6 months ago

njuguna-n commented 6 months ago

Closes #101

njuguna-n commented 6 months ago

There should be an additional PR to update the helm charts if this PR is merged first.

witash commented 6 months ago

with this configuration if the postgrest responds with a 40X response code and doesn't write data. the couchdb plugin will advance the sequence anyway, leading to data loss.

I tested by running locally with docker compose, then dropping the root table. All response from postgrest were then 404 since the table doesn't exist. then run docker exec cht_sync_local-logstash-1 cat /tmp/couchdb/sequence_path.txt several times and it was still increasing.

this also seems to be the case without format => json_batch

I tried every combination of retry_failed, ignorable_codes and automatic_retries but wasn't able to get logstash to stop updating the sequence as long as it got any response from postgrest (success or failure)

It seems the http output plugin differentiates between "library failures" (can't get a connection at all) and "plugin failures" (HTTP status 404) and only stops the pipeline for "library failures" https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-retry_policy

So unless there is a configuration I'm missing, Im not sure the http plugin can work, batch or no batch...

njuguna-n commented 6 months ago

Given the comment above and some other offline discussions about the use of logstash for the sync process I will close this PR.