search

tulip / oplogtoredis

Publish MongoDB oplog entries to Redis
Apache License 2.0
38 stars 9 forks source link

Mongo 5 Support #48

Closed adamtajti closed 1 year ago

adamtajti commented 1 year ago

Copied the changes from the original fork once I realized that they already implemented the processing of v2 messages.

I tried to update the tests too, but it seems like the ./scripts/runIntegrationFaultInjection.sh hangs after these changes, when mongod is being called. I tried to debug it further, but it seems like even mongod --version fails with a hang on the Dockerfile. Maybe it was an ARM specific issue?

jgdef-tulip commented 1 year ago

Before I go through this PR in detail, it seems like the substantial v5 change to the oplog format - and the fact that the oplog is formally undocumented and subject to further arbitrary change without regards to versioning schemes, e.g. semver - is a strong argument that we should change our strategy from tailing the oplog to using change streams.

radekmie commented 1 year ago

Before I go through this PR in detail, it seems like the substantial v5 change to the oplog format - and the fact that the oplog is formally undocumented and subject to further arbitrary change without regards to versioning schemes, e.g. semver - is a strong argumetn that we should change our strategy from tailing the oplog to using change streams.

@jgdef-tulip Do we want to create a ticket for that? I feel like that the ideal eventual architecture shouldn't rely on Mongo -> Redis replication in the long run and those should be done per-service as required. But if we want to depend on this for a long run, then it might be worthwhile. Mongo released on February 2009, we got one major version change in oplog since then. I assume that they won't add a new one it for at least a couple of years to come, while we catch up to Mongo 6 or 7.

As a multi-year oplogtoredis user and promoter, I think at least for now oplog is not changing that much. I improved Meteor's implementation of v2 to v1 converter (including tests; see https://github.com/meteor/meteor/pull/12116) and it was improved only slightly since then (that's how it looks like today). My point is that it is feasible to still use it for this purpose.

However, as I encountered this problem, I knew I'll want to use Change Streams and be certain about the future; it just took too much time to handle it in Meteor and I want to make sure I won't have to do it again 😛

That's why I created changestream-to-redis. Yes, it's basically a drop-in replacement for oplogtoredis but uses Change Streams. At the moment it's used in pre-production environments of the app I'm working on (you can read more about it on my blog) and we're already talking about going to production with it.

As for the scoped name updates, i.e., one.two.three resulting only in one - I skipped this part completely and went with f: [] at all times. It does the exact same thing, as apparently cultofcoders:redis-oplog ignores it anyway. And I can confirm that not sending f: [] just works (we didn't observe any significant performance changes).