current behavior: if elasticsearch goes down, the block processor stops advancing blocks
expected behavior: when ES goes down the block processor should stop cleaning up TouchedOrDeletedPrefixRow entries in the DB (it otherwise only keeps the last 200 blocks). It should also probably stop serving claim_search requests and instead return an error. It should continue advancing blocks without sending anything to ES while it's unavailable.This will allow ES to be down indefinitely. Once it comes back up, the block processor should detect this and on the next update to ES first send the backlog, it should then begin cleaning up the TouchedOrDeletedPrefixRow rows like normal. Additionally, we need a way for it to not lose the notifications to trending, perhaps a new index is needed.
current behavior: if elasticsearch goes down, the block processor stops advancing blocks
expected behavior: when ES goes down the block processor should stop cleaning up
TouchedOrDeletedPrefixRowentries in the DB (it otherwise only keeps the last 200 blocks). It should also probably stop servingclaim_searchrequests and instead return an error. It should continue advancing blocks without sending anything to ES while it's unavailable.This will allow ES to be down indefinitely. Once it comes back up, the block processor should detect this and on the next update to ES first send the backlog, it should then begin cleaning up theTouchedOrDeletedPrefixRowrows like normal. Additionally, we need a way for it to not lose the notifications to trending, perhaps a new index is needed.