Version 4.8.1 will not start due to a segfault.

[ymch]

Currently using version 4.7.2. I have tried updating to versions 4.8.0 and 4.8.1, but neither will start with a segfault. I had no choice but to revert to version 4.7.2.

OS: CentOS Linux 7.9.2009 MongoDB: 4.2.21 / wiredTiger (oplog enable) Node: v14.18.3 NPM: 6.14.15

The /var/log/messages at the time of the error is as follows:

systemd: Started The Rocket.Chat server.
rocketchat: Error creating indexes for integration_history MongoError: Index with name: _updatedAt_1 already exists with different options
rocketchat: at MessageStream.messageHandler (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:272:20)
rocketchat: at MessageStream.emit (events.js:400:28)
rocketchat: at MessageStream.emit (domain.js:475:12)
rocketchat: at processIncomingData (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
rocketchat: at MessageStream._write (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
rocketchat: at writeOrBuffer (internal/streams/writable.js:358:12)
rocketchat: at MessageStream.Writable.write (internal/streams/writable.js:303:10)
rocketchat: at Socket.ondata (internal/streams/readable.js:731:22)
rocketchat: at Socket.emit (events.js:400:28)
rocketchat: at Socket.emit (domain.js:475:12)
rocketchat: at addChunk (internal/streams/readable.js:293:12)
rocketchat: at readableAddChunk (internal/streams/readable.js:267:9)
rocketchat: at Socket.Readable.push (internal/streams/readable.js:206:10)
rocketchat: at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
rocketchat: at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
rocketchat: operationTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1655735991 },
rocketchat: ok: 0,
rocketchat: code: 85,
rocketchat: codeName: 'IndexOptionsConflict',
rocketchat: '$clusterTime': {
rocketchat: clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1655735991 },
rocketchat: signature: { hash: [Binary], keyId: 0 }
rocketchat: }
rocketchat: }
abrt-hook-ccpp: Process 20863 (node) of user 1007 killed by SIGSEGV - dumping core
systemd: rocketchat.service: main process exited, code=dumped, status=11/SEGV
systemd: Unit rocketchat.service entered failed state.
systemd: rocketchat.service failed.

[Fiodin]

Same problem here....

Currently: 4.8.1 Installation: Snap OS: Debian 10

From one minute to the other rocket.chat just stopped working and has the same error as the one above. I think it was as the automatic snap update had run through.

[Fiodin]

I now watched the startup by hitting the service snap.rocketchat-server.rocketchat-server status many times.

Every time the Memory reaches something around 620M it quits and the server is restarted.

I now removed and installed the Snap-Package multiple times and restored backups until 30 days ago. All Backups do have the same problem. When I now restore the last Backup I get the Error: MessageType.render is deprecated. Use MessageType.message instead. livechat_webrtc_video_call

Searching that I get the result that it depends to #25460

My Installation was on 3.12.3 and I upgraded 3 or 4 months ago to the 4.x/stable channel..

My Instance is still not working and I am running out of solutions instead of loosing all the messages and installing a new server.

[DGi-Shadow]

Since automatic Snap upgrade to 4.8.1 Browser based RocketChat and Windows App is no longer working. iOS and Android App is still working. When inspecting the browser I get the following:

Content Security Policy: The settings of the page have blocked the loading of a resource on data:application/octet-stream;base64,AGF... ("connect-src").

Reverting the Server to 3.6.4 is non functional. Before the update server ran fine. Haven't found a solution to this problem.

[firepked]

Since automatic Snap upgrade to 4.8.1 Browser based RocketChat and Windows App is no longer working. iOS and Android App is still working. When inspecting the browser I get the following:

Content Security Policy: The settings of the page have blocked the loading of a resource on data:application/octet-stream;base64,AGF... ("connect-src").

Reverting the Server to 3.6.4 is non functional. Before the update server ran fine. Haven't found a solution to this problem.

I am encountering the same problem. Snap did auto refresh 3days ago.

[Fiodin]

Yesterday I installed the Version 4.1.2 from the latest/edge channel from Snap. The Restoring of the backup was fine but the server still won't start.

I now think that there is a big problem within the data in the database or anywhere inside the data by restoring the backup, In my opinion this was caused by the automatic snap-update.

I will install a new server and will watch if there is a new version available and then give it another try...

[fplante-tink]

Hi, upgrading from 4.7.4 to 5.0.0 also fails to start giving a segfault.

CentOS 7 NodeJS v14.19.2 NPM 6.14.17

[mhaluska]

@fplante-tink There is issue even on RHEL8 I updated my testing instance from 5.0.0 RC12 -> 5.0.0 on RockyLinux 8, latest RC12 works fine.

Jul 22 16:41:00 htz-chat1 rocket.chat[7890]: /opt/rocket.chat/bundle/programs/server/node_modules/fibers/future.js:280
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:                                                 throw(ex);
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:                                                 ^
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]: Error: Can't find migration version 279
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at migrateDatabase (server/lib/migrations.ts:249:9)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at module (server/startup/migrations/xrun.js:8:1)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.require (packages/modules-runtime.js:238:14)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.moduleLink [as link] (/opt/rocket.chat/bundle/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at module (server/startup/migrations/index.ts:1:17)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.require (packages/modules-runtime.js:238:14)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.moduleLink [as link] (/opt/rocket.chat/bundle/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at module (server/startup/index.ts:1:8)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.require (packages/modules-runtime.js:238:14)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.moduleLink [as link] (/opt/rocket.chat/bundle/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at module (server/main.ts:1:29)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Module.require (packages/modules-runtime.js:238:14)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at require (packages/modules-runtime.js:258:21)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at /opt/rocket.chat/bundle/programs/server/app/app.js:216935:15
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at /opt/rocket.chat/bundle/programs/server/boot.js:401:38
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at Array.forEach (<anonymous>)
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at /opt/rocket.chat/bundle/programs/server/boot.js:226:21
Jul 22 16:41:00 htz-chat1 rocket.chat[7890]:     at /opt/rocket.chat/bundle/programs/server/boot.js:464:7

[mddvul22]

We are also having this problem on Centos 7, trying to upgrade from 4.7.5 to 4.8.3.

[shefi]

This still an ongoing issue, even with latest release (5.0.5).

Anything in the works for fixing this bug?

[mddvul22]

I've gotten nowhere even with paid Rocket Chat support. I'm starting to think they don't really care about fixing this issue. We are pondering trying a RHEL 9 server to see if the problem goes away.

[mhaluska]

I've gotten nowhere even with paid Rocket Chat support. I'm starting to think they don't really care about fixing this issue. We are pondering trying a RHEL 9 server to see if the problem goes away.

This makes no sense, CentOS 7 is supported till 2024-06, so why we're using longterm supported OS?

[mddvul22]

I agree 100%.

[mhaluska]

From my point of view it's now not possible to use CentOS7. I tried to build app on CentOS7:

execute yarn in repo root

...
➤ YN0000: ┌ Link step
➤ YN0007: │ fibers@npm:5.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ @rocket.chat/forked-matrix-sdk-crypto-nodejs@npm:0.1.0-beta.12 must be built because it never has been before or the last one failed
➤ YN0009: │ @rocket.chat/forked-matrix-sdk-crypto-nodejs@npm:0.1.0-beta.12 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-4c925bec/build.log)
...

$ cat /tmp/xfs-4c925bec/build.log

# This file contains the result of Yarn building a package (@rocket.chat/forked-matrix-sdk-crypto-nodejs@npm:0.1.0-beta.12)
# Script name: postinstall

/home/rocket/Rocket.Chat/apps/meteor/node_modules/@rocket.chat/forked-matrix-sdk-crypto-nodejs/check-exists.js:15
        throw e;
        ^

Error: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /home/rocket/Rocket.Chat/apps/meteor/node_modules/@rocket.chat/forked-matrix-sdk-crypto-nodejs/lib/index.linux-x64-gnu.node)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1144:18)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/home/rocket/Rocket.Chat/apps/meteor/node_modules/@rocket.chat/forked-matrix-sdk-crypto-nodejs/lib/napi-module.js:145:31)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12) {
  code: 'ERR_DLOPEN_FAILED'
}

CentOS GLIBC 2.17 vs required by app 2.25.

[ymch]

CentOS GLIBC 2.17 vs required by app 2.25.

That issue also occurred in v4.7.0 and I thought it was fixed in v4.7.2, but has it come back? https://github.com/RocketChat/Rocket.Chat/issues/25396#issuecomment-1118442375

We are still using v4.7.4 and are unable to update at all.

[mhaluska]

That issue also occurred in v4.7.0 and I thought it was fixed in v4.7.2, but has it come back? #25396 (comment)

Sorry I forgot to mention: I tried to build latest 5.0.5

[mhaluska]

CentOS GLIBC 2.17 vs required by app 2.25.

Maybe not related, I got same error on 4.7.5 build, but this one is running fine on CentOS 7.

[dudanogueira]

Hi!

Sorry for this, but I believe this is a hard one to solve. It doesn't depend on Rocket.Chat alone, as the cause of this issue is the underlying OS that is running it.

With new versions of Rocket.Chat, new dependencies will eventually come along.

As of now, we have no ETA for a fix on this, but we were discussing some solutions and best approaches internally.

We know it's working on newer OS, so if you can migrate it, that certainly will fix it.

I'm starting to think they don't really care about fixing this issue. I am sorry you feel that way, but that's not true. As you can see on the issue we have about this, we are discussing on how to fix this.

Thank you all for the inputs, we really appreciate it.

[mddvul22]

We gave up waiting and moved to Rocky 8. The upgrade worked on Rocky 8.

[iamfasal]

Any update on this case @dudanogueira

[fcoppolani]

Hi, we tried from 4.7.5 to upgrade to 5.1 with : CentOS 7 NodeJS v14.19.2 NPM 6.14.17

And also fails to start giving a segfault. Unfortunately, we have heavy constraint on the OS (another department manage the OS of all the VMs). On the other side we wanted to upgrade to resolve all security issues opened, would it be possible to patch the 4.7.5 to solve all these security issues ?

CVE-2022-30124, CVE-2022-32217, CVE-2022-32218, CVE-2022-32219, CVE-2022-32220, CVE-2022-32226, CVE-2022-32227, CVE-2022-32228, CVE-2022-32229, CVE-2022-35246, CVE-2022-35247, CVE-2022-35248, CVE-2022-35249, CVE-2022-35250, CVE-2022-35251

Thanks in advance for your help

[iamfasal]

Any update on this case @dudanogueira

[iamfasal]

Is this resolved? @dudanogueira

[cordeosdev]

This is just great... we have a dozen or so Rocket.Chat installations on CentOS 7 and ALL are stuck at version 4.2 to 4.6. With no solution in sight. This is soooooooooo Rocket.Chat!

We have been using, supporting and contributing (testing, code and financially) to Rocket.Chat for over 5 years and it is a CONSTANT upgrade headache. It is a fantastic, solid and reliable platform once you've spent a week pulling your hair out to get it running properly but we hate having to update. Absolutely DREAD it. It is almost always a nightmare. Out of the 10-12 progessive version upgrades we've handled the past 5 years - only one or two have EVER gone smoothly.

Of course, the stock answer is: "just use the docker image!" "all your problems will be solved".

Only developers push docker. Developers who don't really care or understand the other 95% of an organization's IT lifecycle or overall business technology ecosystem. Good luck trying to get your certificate management, compliance tools, security scanning or lifecycle control systems integrated with a docker-based web application! Or if you do... don't worry, it will break very soon, then take weeks to figure out why and fashion a fix.

Docker has a ton of incredibly great use cases... rolling out your scaled-out, tightly secured and highly integrated enterprise production platform is NOT one of them.

There is a very good reason why docker usage is still limited in production enterprise environments. There is also a good reason why most people still have not heard of Rocket.Chat - despite it being an excellent platform (as long as you dont attempt to upgrade it).

So now we are faced with migrating all our CentOS 7 Rocket.Chat clusters to CentOS8... maybe Slack is finally starting to look like a reasonable option.

[RC685]

Also got this when attempting an upgrade from 4.3.1 to whatever is current. Had to restore a snapshot.

Reading through this thread it seems like the issue is unfixable for whatever reason, so I guess we're stuck on this version for now :/

[ymch]

We have over a year until EOL on CentOS7, but I had a feeling that importing chat data in the future would be problematic if I left Rocket.Chat at version 4.7.5. So I moved the server migration timeframe up. The OS of the new server is RockyLinux 9.1. Rocket.Chat installation was done the old way:

dnf install -y nodejs GraphicsMagick npm
npm install -g inherits
npm install -g n
n 14.19.3
cd /opt
curl -L https://releases.rocket.chat/latest/download -o rocket.chat.tar.gz
tar zxvf rocket.chat.tar.gz
mv bundle Rocket.Chat
rm -f rocket.chat.tar.gz
cd Rocket.Chat/programs/server/
npm install
 :
 :

Rocket.Chat Vers: 5.4.1
OS:               Rocky Linux 9.1
MongoDB:          5.0.14 / wiredTiger (oplog enable)
Node:             14.19.3
NPM:              6.14.17

The migration was completed by simply running mongodump to back up the MongoDB data on the old server and then running mongorestore on the new server to import the data.