Make lightblue notification/document event processing truly transactional

[alechenninger]

Scenario:

1. Notification partially processes, producing 100 document events, they all persist (or some portion of them due), but lightblue responds with error or premature EOF
2. We roll back this notification because we don't know if any of the document events were persisted
3. Notification is processed again, now we have as many as 200 document events

That being said, they should mostly be superseded / merged away

[alechenninger]

The heart of this issue is that we want the notification processing to be transactional.

We are almost there, but can do some more. Brainstorming:

1. Retrieve new unprocessed notifications or expired processing ones (done)
2. Update if current to processing (done)
3. Insert document events as pending w/ source notification ids (not done, inserts as unprocessed)
4. Update notifications to processed (done)
5. Update document events to unprocessed (not done, they are already unprocessed)

If 2 fails, stop and start over at 1. If 3 fails, try to rollback via set notifications to unprocessed and delete any pending document events. Then stop and start over at 1. If 4 fails, we could retry. It's either processing or processed, both are recoverable. Stop and start over at 1, unless retry is successful. Do not try to rollback notifications and doc events unless you know they are not marked processed, otherwise you could lose doc events. If 5 fails, we could retry, but that's it. Stop and start over at 1.

For document events, we should retrieve unprocessed (done), old processing (done), as well as expired pending (not done). Expired pending first check if their notifications are processed, and if so these can be processed as well. Otherwise, they can be deleted. We could consider checking if notifications are still processing within some timestamp threshold before deleting. Note that this will probably couple the document event store to the notification store.

1. Retrieve new unprocessed document events, or expired processing, or expired pending.
2. If any expired pending, retrieve their notifications. If notifications are processed, keep. If not, drop. (Can potentially keep if notifications still processing, we wouldn't want to delete the result of an in progress notification, but that would mean the notification was processing for at least as long as pending expiration).
3. Update if current to processing, continue as normal.