search

medic / cht-sync

Data synchronization between CouchDB and PostgreSQL for the purpose of analytics.
GNU General Public License v3.0
2 stars 3 forks source link

80 increase cht sync throughput #82

Closed njuguna-n closed 2 months ago

njuguna-n commented 2 months ago

Closes #80

njuguna-n commented 2 months ago

@witash @lorerod please have a preliminary look as I fix the failing tests and Sonar issues.

witash commented 2 months ago

answering my own question; redis list has a maximum of 2^32 elements, which it will never reach so I guess it will keep queueing until it runs out of disk space; if the logstash plugin can't connect to redis or can't lpush, it doesn't update the sequence and keeps retrying, so that's ok, probably we just need to manage the volume size on production deployments

this branch does depend on the persistent logstash sequence thing...as is, the redis queue is persistent but the logstash sequence isn't, so restarting often leads to a lot of duplicate elements in the queue; it doesn't lead to duplicate data but just has some kind of weird behavior

njuguna-n commented 2 months ago

When I was testing with the BRAC instance logstash was better than postgrest and like you said it led to the redis list becoming bigger and taking up more space. If there is a big discrepancy in the input and output throughputs logstash seems to rate limit itself and decrease the input throughput. Increasing the batch size being sent to postgrest can definitely help in managing this scenario.

njuguna-n commented 2 months ago

The sequence number issue will be addressed in this issue. The final solution might not use redis though.

njuguna-n commented 2 months ago

@lorerod do you mind helping me out with the failing test? All the containers seem to be created okay and the tests all pass locally so not sure what is going on in CI.

njuguna-n commented 2 months ago

Thanks @lorerod. I have renamed the variable to reduce the likelihood of such errors in the future.