Right now, we always start tailing the oplog at the end. This means that if oplogtoredis re-starts, we'll lose all changes that occurred during the restart.
Instead, we should:
Periodically write the timestamp of the last oplog entry that we successfully wrote to Redis to a Redis key
When starting up, if that key exists and is not too far in the past, start from that timestamp instead of the end of the oplog
We'll need to pick some good defaults for the configuration options here (how frequently do we write the latest timestamp to Redis? How far back are we willing to go -- if oplogtoredis has been offline for hours we probably don't want to try to process all the oplog entries, but what's the cutoff? And also allow users to override these defaults.
Right now, we always start tailing the oplog at the end. This means that if
oplogtoredisre-starts, we'll lose all changes that occurred during the restart.Instead, we should:
We'll need to pick some good defaults for the configuration options here (how frequently do we write the latest timestamp to Redis? How far back are we willing to go -- if oplogtoredis has been offline for hours we probably don't want to try to process all the oplog entries, but what's the cutoff? And also allow users to override these defaults.
See: https://github.com/tulip/oplogtoredis/blob/85ebc4bca6b50f572160cdba078c910ebdd89c0b/main.go#L82