Closed Gummikavalier closed 2 years ago
We've got the same problem and similar configuration.
Same error for me too
I get the same error and cannot start with 4.0.0, not RC4.0.0.
The same problem with 4.0.0, upgraded from the latest available 3.18.2 Error: Oct 4 14:47:14 test kernel: node[60646]: segfault at 23d0 ip 00000000000023d0 sp 00007f1cd9abc3c8 error 14 in node[400000+26b5000]
We also have a segfault error after upgrading from 3.18.2 to 4.0.0.
I got the same error upgrading from 3.18.2 to 4.0.0. This was on our self-hosted CentOS 7 server.
Same error here on a Centos 7 self-hosted - needed to downgrade...
Oct 04 16:17:20 chat kernel: node[28685]: segfault at 23d0 ip 00000000000023d0 sp 00007f927df693c8 error 14 in node[400000+267f000]
Same error on self hosted CentOS 7.
Using back previous version (https://releases.rocket.chat/3.18.2/download) works fine.
Using back previous version (https://releases.rocket.chat/3.18.2/download) works fine.
Downgrade worked for me too. Seems upgrading on CentOS 7 self hosted is an issue.
I am having the same issue with a fresh install on Centos and Rocket Chat 4.0.0 self hosted, manual install.
Thanks for all for confirmations! :+1:
Anyone with CentOS Stream or RockyLinux having the same issue? The segfault could be an issue with supported ciphers and dependencies on older OSes.
Tested RC 4.0.1. Still segfaults.
Tested RC 4.0.1. Still segfaults.
Thanks for testing, so I don't have to try it myself. Was just going to ask. :D
Same error on self hosted CentOS 7.
Yeah no 4.x releases are working properly yet.
Thanks for all for confirmations! 👍
Anyone with CentOS Stream or RockyLinux having the same issue? The segfault could be an issue with supported ciphers and dependencies on older OSes.
Rocket.Chat 4.0.1 (upgrade from 3.18.2) fail too with same issue (segfault at 23d0) on CentOS 7.8.2003
Is there any solution for it?
Rocket.Chat 4.0.1 (upgrade from 3.18.2) fail too with same issue (segfault at 23d0) on CentOS 7.8.2003
Is there any solution for it?
Thanks for confirming. No solutions yet.
Same issue here, upgrade from 3.18.1 -> 4.0.1 rhel7 /node-v12.22.5
Can we get a note on the 4.0.0 and 4.0.1 releases warning of this issue for CentOS 7
RC 4.0.2 segfaults still
These may or may not be related to the issue:
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated node-pre-gyp@0.14.0: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future
I think it would be a good idea to peek at the container version of RC4.0 and compare what we have with CentOS7 and its repositories. I'm too busy to do that for a week or two, so I have no choice than waiting for the devs to say something about the issue.
Having the same segfault error on 4.0.2 update.
OS: CentOS Linux release 7.9.2009 (Core) Previous RC version: 3.8.1 Updated to: 4.0.2
The versions below 4.0 seems working fine, so could be a bug in latest release.
v12.22.1
6.14.1
Node and NPM matched as per https://github.com/RocketChat/Rocket.Chat/releases
The service getting killed automatically on startup. The error in kernel:
Oct 15 11:29:51 server.demo.com kernel: node[6071]: segfault at 23d0 ip 00000000000023d0 sp 00007ff0dd01a3c8 error 14 in node[400000+26b5000]
Any solution for this problem in CentOS 7?
@sampaiodiego Any thoughts on this as many users are affected already
Hi.
Please try this
(
cd /opt/Rocket.Chat/programs/server/
rm -rf npm/node_modules/sharp/vendor
NODE_ENV=production npm i
cd ../.. && node main.js
)
I'll update the documentation in a bit.
@debdutdeb Hi, I tried that now (with RC 4.0.2) and get now
$ cd /opt/rocket.chat && node main.js
Something went wrong installing the "sharp" module
libvips-cpp.so.42: cannot open shared object file: No such file or directory
- Remove the "node_modules/sharp" directory, run "npm install" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/en/stable/install/
- Search for this error at https://github.com/lovell/sharp/issues
I'm probably doing something wrong, but I tried couple of different combinations too. I'll wait for the updated docs, unless I figure it out earlier... :)
You need to rebuild the module, i.e. cd programs/server && npm i
I see what I did wrong there, updated the previous comment.
EDIT: made an error again, check now @gummikavalier
Thanks, I'll test
@debdutdeb Whole process I tried now:
Ensure the correct nodejs version:
# node -v
v12.22.1
Ensure the correct NPM version:
# npm install -g npm@6.14.1
# npm cache clean -f
As root remove the old RC path and move untarred RC 4.0.2 bundle to /opt/rocket.chat:
# rm -rf /opt/rocket.chat
# mv /root/bundle /opt/rocket.chat && chown -R rocket:rocket /opt/rocket.chat && chmod -R 770 /opt/rocket.chat && chown -R rocket:rocket /tmp/ufs
Su to rocket account:
# su - rocket
Then first export all the usual working ENV settings (values redacted in below but the ones that worked with RC 3.18.2):
export PORT=
export ROOT_URL=
export MONGO_URL=
export MONGO_OPLOG_URL=
export MAIL_URL=
Then run:
cd /opt/rocket.chat/programs/server/
rm -rf npm/node_modules/sharp/vendor
NODE_ENV=production npm i
cd ../.. && node main.js
Result:
Segmentation fault (core dumped)
Alright, installing centos7 vm. Will write back soon. Thanks
Hello everybody. I'm new and I've never used rocket.chat yet and I'm trying to install it on a centos 7.
I went through the installation process following the guide, I did everything exactly as it is in the documentation about 10 times, always starting from a fresh installation, to make sure I'm not asking for anything.
On the last try I got new errors and I came to this place where people are encountering the exact same problem as me.
I would like to know from friends if there is already something we can do to upload the installation to the centos 7.
Thanks
Jfyi - I'm still investigating the issue. In the meantime, those who are new and trying to install for the first time, you do have other options like docker & snap.
I personally would recommend docker. But snap is a decent option as well. Just install snapd on centos7 (https://snapcraft.io/docs/installing-snap-on-centos) and run
sudo snap install rocketchat-server
Any updates on this? Didn't see anything about it in the 4.0.4 release notes so I assume it's still an issue.
Tested RC 4.0.4 so others don't have to. Still segfaults.
I have the same error when upgrading to version 4.0. Waiting for fix
Same error here, a fix is really needed for those who use CentOS 7. Using snap all works 100%, but I need access to the files for some small changes, so I will wait for a fix until then.
We did some comparing of docker image and tar package, and the differences are so small that it may very well be that the issue is directly at the node binary level. Typically these would be caused by out of date dependencies for crypto or libc.
I've started building a new RHEL8 based server to move our test service config and database to more up to date platform. I'll inform in the comments here how RC 4.0 works on RHEL8.
We really need a fix, for example, I’m running services in CentOS 7, I can’t migrate/upgrade to CentOS 8 because some incompatibility. @debdutdeb any news about this?
Just tried this on 4.1.0 and same issue
Tested on version 4.0.5 same environment than version 3.x and the service crash with a segfault error :
Oct 28 09:35:18 <redacted> kernel: node[3369]: segfault at 23d0 ip 00000000000023d0 sp 00007ffb838d5308 error 14 in node[400000+267f000]
Oct 28 09:35:18 <redacted> abrt-hook-ccpp[3411]: Process 3369 (node) of user 1000 killed by SIGSEGV - dumping core
Oct 28 09:35:24 <redacted> systemd[1]: rocketchat.service: main process exited, code=dumped, status=11/SEGV
Oct 28 09:35:24 <redacted> systemd[1]: Unit rocketchat.service entered failed state.
Oct 28 09:35:24 <redacted> systemd[1]: rocketchat.service failed.
I got my test done, and for those who have the luxury of upgrading their OS, RC 4.0.5 does not segfault in RHEL8.
@Gummikavalier Thanks!
Any ideas on what the issue is in 7? Myself (and probably a lot of other admins of CentOS servers) aren't too keen on upgrading to CentOS 8 since it's going EOL at the end of this year (whereas CentOS 7 will still be supported until 2024).
Anyone tested recent 4.1.0 update? I checked the changelog but I didn't see any mention/fix about this issue.
Hey this looks like an issue at the node binary level :( I haven't been able to figure out the exact source.
A possible (but not ideal) workaround would be to use the node docker image and use an alias since moving fully to docker isn't possible for many of you -
alias node='docker run --rm -e ROOT_URL -e MONGO_URL -e MONGO_OPLOG_URL -e PORT -v $(pwd):/mnt -w /mnt --network host node:12'
alias npm='docker run --rm -v $(pwd):/mnt -w /mnt node:12 npm'
@Gummikavalier Thanks!
Any ideas on what the issue is in 7? Myself (and probably a lot of other admins of CentOS servers) aren't too keen on upgrading to CentOS 8 since it's going EOL at the end of this year (whereas CentOS 7 will still be supported until 2024).
@RC685 I too think that the issue is the node binary. Maybe building node from the sources might help, or not, as the final root cause would likely to be in the libraries it depends on. So your best bet is what debdutdeb suggests above.
But personally I'm moving to RHEL 8 now. I'd move to CentOS Stream or RockyLinux if RHEL8 wasn't available for me.
I see a similar segfault error happened before with RocketChat which is discussed on https://github.com/nodejs/node/issues/19274
I'm still surprised no RocketChat officials addressed this case, and no official fix or solution published yet. The solution @debdutdeb advised using a Docker container for node & npm is not a permanent solution I'd say. Since CentOS 7 EOL extended until 2024, no way we could move the system to CentOS 8 which hits EOL this December and not to CentOS Stream which is a downstream of RHEL going forward. AlmaLinux/RockyLinux is good compared to CentOS 8 or CentOS Stream but we have no way to move to any of these at this stage.
If this is a node binary related error, any chance the higher v12.22 versions would resolved it? I don't see anything related on node package release descriptions re this matter so not sure about it though.
Any permanent solution would be helpful which works with CentOS 7. @sampaiodiego
@fasalsh you're right - the docker solution is definitely not a permanent one. Diego and the engineering team's been busy with the 4.x releases and all the changes coming along with it.
As for me I'm also working on multiple verticals at the same time regarding the latest release as well.
These are the sorts of issues that docker / related technologies are there to solve. Which is why we recommend everyone to implement those solutions to begin with.
That being said nobody's forgot about this issue :) I assure you. @sampaiodiego noticed and notified me about this himself fwiw. My today's already quite stuffed - so I'll dig deeper next week. If this is an issue on the node binary level - 1) there's little for us at rocketchat to do any way 2) it'll take quite a while to get this fixed (which is again not under our control). Thus the workaround of using the node docker image if any of you can't move to docker/k8s for the time being.
You can also use node snap if you don't want to install and configure docker on all those machines - sudo snap install node --channel=12/stable
Or, if you're both against snap and docker usage (I've come across such situations), you can download a minimal ubuntu rootfs archive (since we know node + rc is working fine in ubuntu) and run node through chroot - very similar to using docker tbh.
Again - the above are just temporary workarounds.
@debdutdeb Thanks for the details, and looking forward to a permanent solution for this issue from RocketChat side
version 4.1.1 CentOS Linux 7.9.2009 64bit / Linux 3.10.0-1160.45.1.el7.x86_64 npm 6.14.12 node v12.22.1 mongod v4.0.27
segmentation fault when running
node main.js
Was anyone brave enough to test 4.1.2 already?
Description:
Starting RC node fails with segfault error after upgrading from RC 3.18.1 to RC 4.0.0.
Steps to reproduce:
curl -L https://releases.rocket.chat/latest/download -o /tmp/rocket.chat.tgz
cd /tmp/bundle/programs/server && npm install
Expected behavior:
Node starts and schema upgrade process begins
Actual behavior:
The new node fails to start with a segfault error.
Server Setup Information: