search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.61k stars 10.61k forks source link

5.0.1 breaks message update #26505

Closed kalsan closed 1 year ago

kalsan commented 2 years ago

Description:

When sending and receiving messages, the chat messages no longer update, both in the app and on Desktop. Must refresh the page to see new sent or received messages.

Steps to reproduce:

  1. Go to any channel or PM
  2. Send a message
  3. The message is and remains grey or even invisible definitively
  4. The other side receives a notification, but does not see the new message until the page is refreshed

Expected behavior:

Messages should appear as soon as they are sent or received.

Actual behavior:

Notifications are received, but messages do not appear until reload

Server Setup Information:

Client Setup Information

Relevant logs:

Not sure if relevant, but JS logs give every time a message is sent:

TypeError: 'microphone' (value of 'name' member of PermissionDescriptor) is not a valid value for enumeration PermissionName.
skadefro commented 2 years ago

We are having the same issue

abbluiz-userh commented 2 years ago

Same problem here since 5.0.0

kalsan commented 2 years ago

Still affected: 5.0.2

mphcomputing commented 2 years ago

I have a user seeing the issue on Safari, we are on server version 5.0.1

rtyshyk commented 2 years ago

The same issue after upgrade to v5. Just an idea / it is reproducible after mongo restart. It seems like rocket.chat cannot connect the oplog again to stream updates. After the update to rocket to v5, mongo memory usage increased a lot and OOM kills it very often, after no status updates in rocket.chat.

Romich777 commented 2 years ago

The same issue Chat 5.0.2 Mongo 5.0.5 with oplog

rhulka commented 2 years ago

Same issue after upgrade to 5 (currently 5.0.2 and still present)

franzliedke commented 2 years ago

The new 5.0.3 release seems to have fixed this for us.

kalsan commented 2 years ago

5.0.3 still affected for Firefox 91.12.0esr

wreiske commented 2 years ago

https://docs.rocket.chat/quick-start/deploying-rocket.chat/rapid-deployment-methods/docker-and-docker-compose/docker-containers/high-availability-install#install-mongodb-replicaset

So I had an issue similar to this when upgrading to 4.8 something from 3.8.1. I was able to fix our messages not looking like they had updated by making sure the oploguser had the correct cluster role permission. This happened after upgrading MongoDB to 5 from 3.

db.createUser({user: "oploguser", pwd: "password", roles: [{role: "read", db: "local"}, {role: "clusterMonitor", db: "admin"}]}).

One quick way you may be able to determine if this is your issue is by checking your MongoDB version in the Info section. If it shows "unknown", then you may be affected by this.

image

It should look something like this: image

kalsan commented 2 years ago

Thanks for your suggestion! My output is: 5.0.5 / wiredTiger (oplog Enabled), so this is looking fine.

rtyshyk commented 2 years ago

Updated to 5.0.3, still same issue. mongo db image

dkatsariotis commented 2 years ago

Same issue at Server 5.0.3

kalsan commented 2 years ago

Even worse, now notifications are lost as well. This is getting very frustrating and I am forced to downgrade, as Rocket Chat is effectively unusable in 5.x.

ulope commented 2 years ago

Our users are getting really annoyed about this issue.

This needs to be escalated. If a chat product doesn't show messages this is a big problem.

RC 5.0.4 Mongo 5.0.11 / wiredTiger (oplog Enabled)

skadefro commented 2 years ago

I normally update as soon as rocket.cat sends a message about new releases, so also updated to 5.0.0, and had no issues with that. When updating to 5.0.1 i had the issue too, but I had also just updated my mongodb from 5.x to 6.0.1 so was unsure what broke it, downgrading to 5.0.0 fixed it for me. I updated to 5.5.5 and that seemed to work for about one day, then the issue reappeared and i had do downgrade to 5.0.0 I have just updated to 5.1.0 to see if that is better, but considering the release notes does not mention it, I don't have high hopes. Such a shame the devs are ignorering this.

wreiske commented 2 years ago

I started noticing this on our own server now too. When you send a message or attachment and hit enter, it's like it just vanishes. I retyped the same message and hit send and it worked the second time. After refreshing the client, the message was there twice! Very frustrating.

wreiske commented 2 years ago

Ok - I just asked my contacts at RC and they indeed know about the issue and believe it may be fixed in 5.1. Expect some kind of communication on this thread in the next day or too (please keep in mind it is the weekend).

skadefro commented 2 years ago

And the problem is back, so NOT fixed in 5.1.0

ulope commented 2 years ago

(please keep in mind it is the weekend)

Was it the weekend the last 30 days? RC has aspirations of being a true prime time comms hub but their attitude towards user issues is just negligent.

Time after time real usability issues languish for weeks, months or even years.

himpierre commented 2 years ago

I normally update as soon as rocket.cat sends a message about new releases, so also updated to 5.0.0, and had no issues with that. When updating to 5.0.1 i had the issue too, but I had also just updated my mongodb from 5.x to 6.0.1 so was unsure what broke it, downgrading to 5.0.0 fixed it for me. I updated to 5.5.5 and that seemed to work for about one day, then the issue reappeared and i had do downgrade to 5.0.0 I have just updated to 5.1.0 to see if that is better, but considering the release notes does not mention it, I don't have high hopes. Such a shame the devs are ignorering this.

I'm confused. I'm reading the problem does not occur with mongodb 5.0.x. Is that right?

jhfrontz commented 2 years ago

I'm even more confused -- there was a comment by someone (@debdutdeb?) a few days ago that has since disappeared:

image

skadefro commented 2 years ago

No, others using mongodb 5 also have the issue. And no, it is not fixed in rocketchat 5.1.0

himpierre commented 2 years ago

Just to be clear. This happens to every single message you send @skadefro? Or is it one of ten? By the way, I do not see such behaviour on a few of my instances but the plan is to upgrade our production chat to 5.1.1 soon. So I'm very interested. :)

skadefro commented 2 years ago

Just to be clear. This happens to every single message you send @skadefro?

Hard to say. It seems to go away if you restart rocket chat, but then comes back after a while. last time it worked for 1½ day I would hate to be the dev trying to find the issue, nothing worse than an issue that is not consistent.

jhfrontz commented 2 years ago

Just to be clear. This happens to every single message you send @skadefro? Or is it one of ten?

For me, it's essentially all the time. If I refresh the (Safari) window, it will show the missing content and perhaps will show new content for a few exchanges. But it inevitably freezes again (necessitating a browser-window refresh). Interestingly, even when the window stops updating, I still get Safari notifications when new content is sent (by others) to the affected channel -- and others see anything that I send (even though I can't see what I sent -- until I refresh the browser-window).

himpierre commented 2 years ago

So we're talking about Safari here? Are other browsers affected as well?

Am 8. September 2022 20:01:11 MESZ schrieb Jeff Frontz @.***>:

Just to be clear. This happens to every single message you send @skadefro? Or is it one of ten?

For me, it's essentially all the time. If I refresh the (Safari) window, it will show the missing content and perhaps will show new content for a few exchanges. But it inevitably freezes again (necessitating a browser-window refresh). Interestingly, even when the window stops updating, I still get Safari notifications when new content is sent (by others) to the affected channel -- and others see anything that I send (even though I can't see what I sent).

skadefro commented 2 years ago

The electron app on windows, chrome on windows, the app on ios and chrome on ios ( so safari )

himpierre commented 2 years ago

Strange. Usually this kind of behaviour comes from mongodb connection problems. Something in the logs? Did you try to raise the debug level if nothing shows up?

Am 8. September 2022 20:23:14 MESZ schrieb Allan Zimmermann @.***>:

The electron app on windows, chrome on windows, the app on ios and chrome on ios ( so safari )

skadefro commented 2 years ago

I have auto scaling enabled, ( using mongodb atlas ) it might have triggered after at up/down scale some of the other times, but since last restart of my rocket chat, it has not scaled up or down

himpierre commented 2 years ago

Can you reproduce the error on https://open.rocket.chat/home?

Am 8. September 2022 20:36:23 MESZ schrieb Allan Zimmermann @.***>:

I have auto scaling enabled, ( using mongodb atlas ) it might have triggered after at up/down scale some of the other times, but since last restart of my rocket chat, it has not scaled up or down

skadefro commented 2 years ago

Just tested from all 4 clients, had no issues.

jhfrontz commented 2 years ago

Can you reproduce the error on https://open.rocket.chat/home? Am 8. September 2022 20:36:23 MESZ schrieb Allan Zimmermann @.***>: …

After switching to another tab for a few minutes and then switching back:

image

kalsan commented 2 years ago

Just to be clear. This happens to every single message you send @skadefro? Or is it one of ten?

For me, it's essentially all the time. If I refresh the (Safari) window, it will show the missing content and perhaps will show new content for a few exchanges. But it inevitably freezes again (necessitating a browser-window refresh). Interestingly, even when the window stops updating, I still get Safari notifications when new content is sent (by others) to the affected channel -- and others see anything that I send (even though I can't see what I sent -- until I refresh the browser-window).

Same problems under Firefox, Chrome and the Android App. Standard Docker setup as recommended on the documentation, no scaling or other fanciness.

ulope commented 2 years ago

For us a handful of users out of ~40 have reported the issue. Both in the browser and the electron app.

debdutdeb commented 2 years ago

Is this happening still on 5.1.x? We haven't seen a single instance of it since then. Please confirm (possibly a screen recording as well).

dkatsariotis commented 2 years ago

Is this happening still on 5.1.x? We haven't seen a single instance of it since then. Please confirm (possibly a screen recording as well).

I confirm that the issue still remains on 5.1.1 Using Windows client and web browser.

Windows pop up notifications work but no new message update in chat window.Only by using Ctrl+R we get it work until new message arrives.

debdutdeb commented 2 years ago

Can you share a screen recording please?

rtyshyk commented 2 years ago

Can you share a screen recording please?

@debdutdeb The chat client does not matter, the issue is reproducible in mobile/web/desktop clients. messages not received automatically, you have to refresh page manually. example below shows status updates, but the same issue when someone receives issue, you don't see it until the page is refreshed.

Looks like after mongo restart, rocket.chat cannot connect to the oplog again to receive and push updates to socket. when page is rendered, it takes data from db and everything is fine.

https://user-images.githubusercontent.com/1192326/189412589-2e905a92-1f72-42b5-a4b9-7fac0eed570d.mp4

wreiske commented 2 years ago

Can you share a screen recording please?

@debdutdeb The chat client does not matter, the issue is reproducible in mobile/web/desktop clients. messages not received automatically, you have to refresh page manually. example below shows status updates, but the same issue when someone receives issue, you don't see it until the page is refreshed.

Looks like after mongo restart, rocket.chat cannot connect to the oplog again to receive and push updates to socket. when page is rendered, it takes data from db and everything is fine.

screencast.2022-09-09.20-40-06.mp4

That's actually different behavior than what I've experienced. When I send a message, it just disappears. No grayed out text. Just as if the message was never sent at all. After refreshing it shows up. I haven't been able to reproduce while screen recording.

jhfrontz commented 2 years ago

That's actually different behavior than what I've experienced. When I send a message, it just disappears. No grayed out text. Just as if the message was never sent at all. After refreshing it shows up. I haven't been able to reproduce while screen recording.

Same here -- though every-so-often, I might see one (at most) of my sent messages grayed out. And I see zero messages from other participants in the channel (though I do sometimes see indications that they are typing -- as well as their content via the MacOS notification center).

dkatsariotis commented 2 years ago

Can you share a screen recording please?

Hi, sorry for the delay

https://user-images.githubusercontent.com/22136058/189594747-3503ba00-f86f-4e37-8982-a17001366a14.mp4

sandrotanner commented 2 years ago

@debdutdeb any news from the team on this?

jhfrontz commented 2 years ago

Another interesting observation -- if I'm scrolled back in a channel's history and new messages are sent (by others) to the channel, I will (in addition to the macos notification) get a little "new message" arrow-button at the bottom. Clicking on it (or scrolling to the bottom) scrolls to the bottom but does not reveal the new messages.

FlutterFraz commented 2 years ago

I am suffering from the same problem

I must say that i can receive notifications, but the messages appear only once i refresh the page

lestercoyoyjr commented 2 years ago

Hello everyone,

Depending on your deployment but at least for those who are in docker, can you try this:

  1. first back up your DB and take everything down via: docker-compose down
  2. Then, start them up one by one in this order:
    • docker-compose up -d mongo - give it a minute or two to start
    • docker-compose up -d mongo-init-replica - give this one a minute to start as well.
  3. Then finally: docker-compose up -d should bring the rest of the stack up.
HrHeun commented 2 years ago

The same problem may be causing the issues #26820 and #26844

skadefro commented 2 years ago

It has been one and a half month with this bug, plaguing several users. Is there a fix in the works or does this not have priority?

Vaxter commented 2 years ago

Looks like not enough of us are affected, so this has no priority whatsoever. I'll give it another week or so and then I am migrating away from Rocket.Chat.

wreiske commented 2 years ago

I am running:

Version
5.1.2
Apps Engine Version
1.34.0
Node Version
v14.19.3
Database Migration
281 (September 23, 2022 12:21 PM)
MongoDB
5.0.9 / wiredTiger (oplog Enabled)

With 4 instances of Rocket.Chat, nginx as a proxy in front, and I have not had this issue recently across almost 100 users.

Has anyone had this issue on 5.1.2? I'm wondering if it's related to a rate-limiting issue?

image

I have all rate limits turned OFF. Maybe check your rate limit settings?

Also, if you skipped over it, please check this comment https://github.com/RocketChat/Rocket.Chat/issues/26505#issuecomment-1213651710 and make sure that your MongoDB user has the correct permissions.

Take a look at your access logs during the time the issues arise. Are you seeing failures in your web server access logs? (nginx, etc?)

Check the Rocket.Chat service logs, do you see anything that stands out there?

What Rocket.Chat Desktop or Mobile App versions are your users using? I had an issue that was completely resolved by upgrading to the latest Desktop on Windows.

(please remember I am only a community member, I do not work for Rocket.Chat, but I am trying to help and get attention on this issue for you all!)