Think about using RedisOplog & Grapher

[theodorDiaconu]

And make rocketchat the most scalable chat ever:

https://github.com/cult-of-coders/redis-oplog https://github.com/cult-of-coders/grapher

What do you think ? Love this initiative a lot. We invested a lot in making Meteor scalable

[AFrangopoulos]

@theodorDiaconu I am trying (and failing) to get redis-oplog to work on my RC Instance. Is rocket chat's current infrastructure blocking the ability to use redis-oplog? I was able to easily add redisOplog to a simple app I made to test it out, but not able to do so for my RC app.

[theodorDiaconu]

@AFrangopoulos what're the errors ?

[AFrangopoulos]

Application starts successfully, and it does a bunch of dummy collection sub/unsubs. Once I try to go to the application URL (localhost:3000), i get this trace (and app crashes):

I20180329-11:00:39.995(-4)? [1522335639994] - [Main] New incomming subscription for publication: null
I20180329-11:00:40.033(-4)? [1522335640032] - [Main] New incomming subscription for publication: userData
I20180329-11:00:40.038(-4)? Exception in callback of async function: TypeError: res[i]._getCollectionName is not a function
I20180329-11:00:40.038(-4)?     at Subscription._publishHandlerResult (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1249:37)
I20180329-11:00:40.038(-4)?     at Subscription._runHandler (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1200:10)
I20180329-11:00:40.039(-4)?     at Subscription.subscriptionProto._runHandler (packages/meteorhacks_kadira.js:2781:24)
I20180329-11:00:40.039(-4)?     at Session._startSubscription (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1024:9)
I20180329-11:00:40.039(-4)?     at Session.sub (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:816:12)
I20180329-11:00:40.039(-4)?     at packages/meteorhacks_kadira.js:2644:34
I20180329-11:00:40.039(-4)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
I20180329-11:00:40.039(-4)?     at Session.sessionProto.protocol_handlers.sub (packages/meteorhacks_kadira.js:2643:44)
I20180329-11:00:40.040(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:754:85
I20180329-11:00:40.040(-4)? 
W20180329-11:00:40.171(-4)? (STDERR) Uncaught exception: TypeError: res[i]._getCollectionName is not a function
W20180329-11:00:40.172(-4)? (STDERR)     at Subscription._publishHandlerResult (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1249:37)
W20180329-11:00:40.172(-4)? (STDERR)     at Subscription._runHandler (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1200:10)
W20180329-11:00:40.172(-4)? (STDERR)     at Subscription.subscriptionProto._runHandler (packages/meteorhacks_kadira.js:2781:24)
W20180329-11:00:40.172(-4)? (STDERR)     at Session._startSubscription (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1024:9)
W20180329-11:00:40.173(-4)? (STDERR)     at Session.sub (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:816:12)
W20180329-11:00:40.173(-4)? (STDERR)     at packages/meteorhacks_kadira.js:2644:34
W20180329-11:00:40.173(-4)? (STDERR)     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
W20180329-11:00:40.173(-4)? (STDERR)     at Session.sessionProto.protocol_handlers.sub (packages/meteorhacks_kadira.js:2643:44)
W20180329-11:00:40.174(-4)? (STDERR)     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:754:85
W20180329-11:00:40.174(-4)? (STDERR) 
W20180329-11:00:40.252(-4)? (STDERR) TypeError: res[i]._getCollectionName is not a function
W20180329-11:00:40.252(-4)? (STDERR)     at Subscription._publishHandlerResult (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1249:37)
W20180329-11:00:40.252(-4)? (STDERR)     at Subscription._runHandler (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1200:10)
W20180329-11:00:40.258(-4)? (STDERR)     at Subscription.subscriptionProto._runHandler (packages/meteorhacks_kadira.js:2781:24)
W20180329-11:00:40.259(-4)? (STDERR)     at Session._startSubscription (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:1024:9)
W20180329-11:00:40.259(-4)? (STDERR)     at Session.sub (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:816:12)
W20180329-11:00:40.259(-4)? (STDERR)     at packages/meteorhacks_kadira.js:2644:34
W20180329-11:00:40.259(-4)? (STDERR)     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
W20180329-11:00:40.259(-4)? (STDERR)     at Session.sessionProto.protocol_handlers.sub (packages/meteorhacks_kadira.js:2643:44)
W20180329-11:00:40.260(-4)? (STDERR)     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/ddp-server.js:754:85```

[theodorDiaconu]

You did not add it as the first package as the instructions said.

[AFrangopoulos]

🤦‍♂️ Ah I forgot about that in the notes. After doing that I get another Exception. App isn't "crashed" but when accessing the application it just spins forever.

I20180329-13:12:31.975(-4)? Exception in callback of async function: Error: [When you subscribe directly, you can't have other specified fields rather than $in]
I20180329-13:12:31.975(-4)?     at packages/cultofcoders:redis-oplog/lib/utils/extractIdsFromSelector.js:9:19
I20180329-13:12:31.975(-4)?     at RedisSubscriber.getChannels (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:39:29)
I20180329-13:12:31.975(-4)?     at new RedisSubscriber (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:22:30)
I20180329-13:12:31.975(-4)?     at PublicationEntry.init (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:50:32)
I20180329-13:12:31.976(-4)?     at new PublicationEntry (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:22:14)
I20180329-13:12:31.976(-4)?     at PublicationFactory.create (packages/cultofcoders:redis-oplog/lib/cache/PublicationFactory.js:44:32)
I20180329-13:12:31.976(-4)?     at createPublicationEntry (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:37:33)
I20180329-13:12:31.976(-4)?     at Cursor.observe (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:25:20)
I20180329-13:12:31.976(-4)?     at Object.observe (packages/rocketchat:lib/server/models/_BaseCache.js:688:45)
I20180329-13:12:31.976(-4)?     at inject.js (packages/rocketchat:ui-master/server/inject.js:26:81)
I20180329-13:12:31.976(-4)?     at fileEvaluate (packages/modules-runtime.js:343:9)
I20180329-13:12:31.976(-4)?     at require (packages/modules-runtime.js:238:16)
I20180329-13:12:31.977(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/rocketchat_ui-master.js:386:1
I20180329-13:12:31.977(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/rocketchat_ui-master.js:391:3
I20180329-13:12:31.977(-4)?     at infos.forEach.info (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:418:13)
I20180329-13:12:31.977(-4)?     at Array.forEach (<anonymous>)
I20180329-13:12:31.977(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:417:9
I20180329-13:12:31.977(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:471:5
I20180329-13:12:31.977(-4)?     at Function.run (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/profile.js:510:12)
I20180329-13:12:31.978(-4)?     at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:470:11

Inject.js:26 is:

RocketChat.models.Settings.find({_id:/theme-color-rc/i}, {fields: { value: 1}}).observe({
    changed: renderDynamicCssList
});

[theodorDiaconu]

Oh... never thought about this usecase of a regex id. Pls file an issue on redisoplog thnx

Regards, Theodor

On 29 Mar 2018, at 8:34 PM, Aleko Frangopoulos notifications@github.com wrote:

🤦‍♂️ Ah I forgot about that in the notes. After doing that I get another Exception. App isn't "crashed" but when accessing the application it just spins forever.

I20180329-13:12:31.975(-4)? Exception in callback of async function: Error: [When you subscribe directly, you can't have other specified fields rather than $in] I20180329-13:12:31.975(-4)? at packages/cultofcoders:redis-oplog/lib/utils/extractIdsFromSelector.js:9:19 I20180329-13:12:31.975(-4)? at RedisSubscriber.getChannels (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:39:29) I20180329-13:12:31.975(-4)? at new RedisSubscriber (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:22:30) I20180329-13:12:31.975(-4)? at PublicationEntry.init (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:50:32) I20180329-13:12:31.976(-4)? at new PublicationEntry (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:22:14) I20180329-13:12:31.976(-4)? at PublicationFactory.create (packages/cultofcoders:redis-oplog/lib/cache/PublicationFactory.js:44:32) I20180329-13:12:31.976(-4)? at createPublicationEntry (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:37:33) I20180329-13:12:31.976(-4)? at Cursor.observe (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:25:20) I20180329-13:12:31.976(-4)? at Object.observe (packages/rocketchat:lib/server/models/_BaseCache.js:688:45) I20180329-13:12:31.976(-4)? at inject.js (packages/rocketchat:ui-master/server/inject.js:26:81) I20180329-13:12:31.976(-4)? at fileEvaluate (packages/modules-runtime.js:343:9) I20180329-13:12:31.976(-4)? at require (packages/modules-runtime.js:238:16) I20180329-13:12:31.977(-4)? at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/rocketchat_ui-master.js:386:1 I20180329-13:12:31.977(-4)? at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/packages/rocketchat_ui-master.js:391:3 I20180329-13:12:31.977(-4)? at infos.forEach.info (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:418:13) I20180329-13:12:31.977(-4)? at Array.forEach () I20180329-13:12:31.977(-4)? at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:417:9 I20180329-13:12:31.977(-4)? at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:471:5 I20180329-13:12:31.977(-4)? at Function.run (/Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/profile.js:510:12) I20180329-13:12:31.978(-4)? at /Users/afrangopoulos/Development/Rocket.Chat/.meteor/local/build/programs/server/boot.js:470:11 Inject.js:26 is:

RocketChat.models.Settings.find({_id:/theme-color-rc/i}, {fields: { value: 1}}).observe({ changed: renderDynamicCssList }); — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[AFrangopoulos]

Created issue. Thanks for your help Theodor!

[AFrangopoulos]

@theodorDiaconu I removed the problematic query just to see if was successful afterward -- and I am still receiving the original error from my first stack trace (just further along in the sub/unsub process). I put the redis-oplog package name at the top of list in .meteor/packages and re-built/ran the meteor app.

Is there something else I am missing or is this still the same root cause? I am not sure what else I would need to do to ensure redisOplog is the "first option"

[theodorDiaconu]

Can you create a repo? I can take a look at it in the next days.

Regards, Theodor

On 29 Mar 2018, at 9:30 PM, Aleko Frangopoulos notifications@github.com wrote:

@theodorDiaconu I removed the problematic query just to see if was successful afterward -- and I am still receiving the original error from my first stack trace (just further along in the sub/unsub process). I put the redis-oplog package name at the top of list in .meteor/packages and re-built/ran the meteor app.

Is there something else I am missing or is this still the same root cause? I am not sure what else I would need to do to ensure redisOplog is the "first option"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[AFrangopoulos]

I could but I'm not sure it's necessary. I just pulled the latest RC (sometime this week) and then I added meteorhacks:kadira and meteorhacks:meteor-down. But I also tested the repo without those packages and still got the error.

Therefore, if you pull latest Rocket Chat repo and just add redisOplog/disable oplog packages and then modify the .meteor/packages file to have redisOplog at the top. That is what I have.

Update: You will also need to comment out that line from above from injest.js that has the regex _id case redis has not accounted for. If we can get passed these errors, I will be working with this quite a bit and will be testing as well.

[AFrangopoulos]

@theodorDiaconu Any luck with having some time?

@geekgonecrazy Have any RC devs tried integrating redisOplog instead of using oplog tailing?

[geekgonecrazy]

Redis oplog is definitely on our radar. We've been watching it develop, especially with the recent flurry of activity. :smile:

We have not begin much testing of it. Introducing a new dependency like redis is not something we want to rush into. We have a lot of users, if upgrading suddenly required them to setup redis... it would be rather complicated to work through :smile:

Would love to see some new numbers as to the performance gains of using redis-oplog instead of regular oplog.

[theodorDiaconu]

@AFrangopoulos very likely I will have time this week to look into the issue.

@geekgonecrazy

We have not begin much testing of it. Introducing a new dependency like redis is not something we want to rush into. We have a lot of users, if upgrading suddenly required them to setup redis... it would be rather complicated to work through 😄

This solution should act as an opt-in, for enterprise features, where scalability is required. RedisOplog for 50-100 online users that chat at a rate of 20 msgs / minute, it's useless, you're better off with oplog.

RedisOplog is more of a safety net for apps that grow, and they eventually hit the oplog bottleneck, they have the ability to switch with minor changes in their code-base.

[geekgonecrazy]

@theodorDiaconu makes sense. So you see biggest gains in connected users? Would be great to have a more in depth conversation to find out where exactly these gains are coming from. To see if some of our own performance pains could be relieved. I suspect they would, but important to really understand how and why that's the case.

[theodorDiaconu]

@geekgonecrazy it's not the number of users that is important, it's the actual traffic. Scoop a bit through the docs of RedisOplog, everything is explained there, if you do have other questions feel free to ask. On the repository.

[AFrangopoulos]

@theodorDiaconu My email is AFrangopoulos@updox.com . If we could start an email thread that would be better/more efficient. Any information that is pertinent to RC users trying to get redisOplog working I can always post here instead of filling up this Issue board.

[kaiiiiiiiii]

Why not using the rocket.chat forum for that @AFrangopoulos ?

[AFrangopoulos]

I had some off topic questions but I could still post there as well.

[AFrangopoulos]

Just a little update: We were able to spend an hour or so on this issue and got a little bit further. It seems like the livechat/livestream packages were an issue so we removed them. Then we were able to at least reach the login page. From there, the userRoles collection is the next obstacle. This isn't the best approach (tearing away RC features to get something working), but at this point we just wanted to try to get by everything as quickly as possible in order to run some messaging tests with RC. So far, we are still stuck at login page and the userRoles collection error for _getCollectionName() as shown above.

If we make more constructive progress I will update. Otherwise, any help from @theodorDiaconu or RC devs would be greatly appreciated. Thanks!

[rodrigok]

We are developing a micro-services architecture and our streamer component will solve this problem being a single piece connected on mongodb via query streams.

So I'll close this since we don't have plans to support redisOplog.

[ndroo]

Did the streamer ever eventuate? I have a need for redisoplog given a specific infrastructure setup.

I did explore this further and managed to load the server to load and I can send messages, but its not working as it's not pulling the oplog events as I was expecting (the symptom is that messages stay grey and only turn black after a refresh).

All i had to do was

1. Install redisoplog as per the instructions
2. In server/startup/serverRunning.js comment out "exitIfNotBypassed(process.env.BYPASS_OPLOG_VALIDATION);"
3. Start the server

It warns that the oplog is disabled, but i guess thats kind of the point of using redisoplog? anyway, ill play more, but its not working yet.