antkaz commented 3 years ago

Description:

When a user changes username in chat, all messages in the collection are updated. This places a heavy load on the database if there are many documents in the collection.

rocketchat_message collection size: ~88 GB, 57 million documents

CPU usage

Steps to reproduce:

For reproduce needs a lot of documents in the collection rocketchat_message and a user should have a lot of messages

Go to profile
change username

Expected behavior:

Chat works

Actual behavior:

Chat is not available

Server Setup Information:

Version of Rocket.Chat Server: 3.6.3
Operating System: Debian
Deployment Method: Docker
Number of Running Instances: 4
DB Replicaset Oplog: enabled
MongoDB Version: 4.0

Relevant logs:

Logs from DB

update rocketchat.rocketchat_message command: { q: { u._id: "kPzqtR7ahm5z5Wgtx" }, u: { $set: { u.username: "Kerry", _updatedAt: new Date(1606316642299) } }, multi: true, upsert: false } planSummary: IXSCAN { u._id: 1 } keysExamined:52516 docsExamined:52516 nMatched:52516 nModified:52516 keysInserted:105032 keysDeleted:105032 numYields:7797 locks:{ Global: { acquireCount: { r: 7798, w: 7798 } }, Database: { acquireCount: { w: 7798 }, acquireWaitCount: { w: 4 }, timeAcquiringMicros: { w: 105158 } }, Collection: { acquireCount: { w: 7798 } } } storage:{ data: { bytesRead: 3349435742, timeReadingMicros: 64408232 } } 97265ms

mmgoilf commented 3 years ago

We have this same issue, changing username temporarily breaks the entire server. We are on version 3.9.1, but this has been a ongoing issue for as long as I can remember.

sampaiodiego commented 3 years ago

there are two issues happening here:

since we're using a non-relational database, we decided to copy the username into the message object. so if you change it, we need to update all messages you have sent. this is a common trade off when using non-relational databases but we're looking into how to improve it
because rocket.chat keeps listening to mongodb events, if such update is performed on mongodb and changes too much records, rocket.chat will be notified of EVERY SINGLE record changed, which will eventually cause the process to halt.

we're looking into probably removing the username from the message object and the frontend will request them when needed. this will fix both issues, but we don't have an expected date to make this though

cc @rodrigok @ggazzo

antkaz commented 3 years ago

@sampaiodiego thanks for your answer. Where is the u.username field used? Can I query data from the "users" collection through aggregation?