Messages sending out of order (or, frequently, not at all)

DarkRider1768 commented 5 years ago

Description:

Messages are displaying out of order, if they display at all.

Steps to reproduce:

Send a private message.
Anticipate it showing on the next line below.
Observe it showing up several lines above.

Expected behavior:

Send a private message. Observe it showing on the next line below.

Actual behavior:

Send a private message. Observe it showing up several lines above.

Server Setup Information:

Version of Rocket.Chat Server: 0.74.3
Operating System: Docker\Kubernetes
Deployment Method: Kubernetes
Number of Running Instances: 4
DB Replicaset Oplog: Yes
NodeJS Version: Baked into image
MongoDB Version: 3.6

Additional context

We've had nothing but issues with our Rocketchat install. Notifications not being sent, having to constantly reload without cache to be able to see messages, messages that don't show up for weeks...

Some of our config:

    spec:
      containers:
      - env:
        - name: MAIL_URL
          value: smtp://ehs-*****.****.ad
        - name: MONGO_OPLOG_URL
          value: mongodb://10.***.***.41:27017,10.***.***.42:27017,10.***.***.43:27017,10.***.***.44:27017/local?replicaSet=rs0
        - name: MONGO_URL
          value: mongodb://10.***.***.41:27017,10.***.***.42:27017,10.***.***.43:27017,10.***.***.44:27017/rocketchat?replicaSet=rs0&readPreference=nearest&w=majority
        - name: PORT
          value: "3000"
        - name: ROOT_URL
          value: http://ehs-******.******.ad
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        image: rocketchat/rocket.chat:latest
        imagePullPolicy: IfNotPresent
        name: rocketchat
        ports:
        - containerPort: 3000
          hostPort: 3000
          name: 3000tcp30000
          protocol: TCP
        - containerPort: 3000
          name: 3000tcp803
          protocol: TCP
        - containerPort: 9100
          name: 9100tcp2
          protocol: TCP

Port 3000 to 3000 is a host port to allow inter-server communication. Port 3000 to 80 is a LoadBalancer port to allow load balanced access from the clients (and to enable HA). Port 9100 is a cluster IP port to allow Prometheus scrapes for metrics.

Relevant logs:

We are only consistently getting a single type of error message in our logs since disabling DDP rate limiting (which prevented us from using Rocketchat at all):

Error in oplog callback TypeError: Cannot read property 'u' of undefined
at BaseDb.RocketChat.models.Subscriptions.on (server/publications/subscription.js:84:50)
at emitOne (events.js:116:13)
at BaseDb.emit (events.js:211:7)
at BaseDb.processOplogRecord (packages/rocketchat:models/server/models/_BaseDb.js:143:9)
at packages/mongo/oplog_tailing.js:105:7
at runWithEnvironment (packages/meteor.js:1356:24)
at Object.callback (packages/meteor.js:1369:14)
at packages/ddp-server/crossbar.js:114:36
at Array.forEach (<anonymous>)
at Function._.each._.forEach (packages/underscore.js:139:11)
at DDPServer._Crossbar.fire (packages/ddp-server/crossbar.js:112:7)
at handleDoc (packages/mongo/oplog_tailing.js:311:24)
at packages/mongo/oplog_tailing.js:337:11
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1304:12)
at packages/meteor.js:620:25
at runWithEnvironment (packages/meteor.js:1356:24)

fsejoseph commented 5 years ago

I am having the same issue. Any luck on a solution?

DarkRider1768 commented 5 years ago

No, unfortunately. It comes and it goes. We updated to 1.0.0-rc3 in the hopes that it might have tackled some of the issues but it's too early to tell if it fixed that particular issue.

brittyazel commented 4 years ago

We to have this issue on server version 2.3.0. on Android consistently newly sent messages get placed before the previously received messages. Only after leaving that channel and returning will the messages be in the right order

reetp commented 4 years ago

We to have this issue on server version 2.3.0. on Android consistently newly sent messages get placed before the previously received messages. Only after leaving that channel and returning will the messages be in the right order

Which version of Android?

brittyazel commented 4 years ago

I am using Android 9 (OnePlus 5t). I am using RocketChat Experimental 1.25.0.13048 and the server is on 2.3.0.

This happens every single time I send messages. The easiest way to replicate it is in sending 2+ messages back to back. Every message after the first will get positioned ABOVE the older messages rather than below. It's like it's ordering them backwards. And, like I said before, leaving the channel and returning will have the messages put in the proper order.

ankar84 commented 4 years ago

I'am sure, that I am having the same issue with RC Version 2.3.1 in RC.Electon Version 2.16.0 But I think I saw it first time today. We have 3 RC Running Instances with OpLog Enabled. And now I see one strange thing - Mongo Storage Engine shows unknown instead of MMAP

ankar84 commented 4 years ago

Probably Mongo Storage Engine shows unknown not related to that issue. It become unknown when our administrator configured MongoDB with authorization. But today I saw problem described in that issue again. My messages appears upper the interlocutor message in dialog even though I sent a message later.

ankar84 commented 4 years ago

@DarkRider1768 and @fsejoseph I think that I found root cause for that issue. It was time! We have 3 instances and there was about 10 seconds difference between then. After we configured all nodes to single NTP server problem was solved. Can you check time accuracy on all of your 4 instances?

brittyazel commented 4 years ago

Yep it was the server clock. It was behind by about 40s in our case

RocketChat / Rocket.Chat