Replace Logstash and Redis with an updated version of couch2pg

[dianabarsan]

Logstash, in combination with logstash-input-couchdb_changes, is not production ready for a variety of reasons:

1. It’s unmaintained: last commit was 8 years ago
2. Does not follow CouchDb replication protocol by only saving the seq (checkpointer) on the source, and not on the target. According to the CouchDb replication protocol, the sequence must be saved on the source and target, to cover situations where the target is rolled back.
3. Does not follow replication protocol, and advances the sequence immediately after queueing the change, instead of waiting for the write to happen successfully.
4. Only uses basic authentication One of our biggest efforts this year is to switch to cookie authentication for all services in order to allow us to increase password hashing iterations.
5. Looks like it has dead and unused code, like an ignore_attachments config
6. Very low coverage happy path only tests
7. Has a bug about not being able to connect to a secure CouchDb server

A good alternative is to bring a refactored version of couch2pg to replace this part of the stack. The updates required (not exhaustive):

• Update dependencies:
  • consider using node-postgres instead of pg-promise
  • update postgrator (we are behind ~5 majors)
  • update PouchDb and use pouchdb-session-authentication directly
• Update NodeJS
• Sync output with cht-pipeline (add @timestamp for example)
• Use current JS code standards
• Update so it does not delete old revs of docs that have changed (under debate)
• Make it run continuously and be resilient against runtime errors
• Dockerize

Proposal doc: https://docs.google.com/document/d/1qqXQ-O9Fimn6UvVQUf5BsOF96caoK1xo0uYZbXzYdSs/edit

[njuguna-n]

@dianabarsan I have a couple of hours free and would like to help with this task. How can I help? Can I set up separate tickets for the updates required and pickup one of them?

[dianabarsan]

I'm not entirely sure how I would break this appart. I'll let you know if I think of something :)

[dianabarsan]

You could definitely help by adding e2e tests, which are not blocked by this work!

[andrablaj]

FYI, there is already a ticket for testing opened by Lorena, so you can link any testing work to it.