search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.4k stars 10.51k forks source link

Crash on startup after upgrade to 0.62.2 #10114

Closed micw closed 6 years ago

micw commented 6 years ago

Description:

I have been trying to upgrade my docker installation from 0.58.4 to 0.62.2. On a direct upgrade, Rocket.Chart crashed shortly after start. So I did a rollback and a step-by-step upgrade from 0.58.4 to 0.59.6 to 0.60.4 to 0.61.2 to 0.62.2. Up to 0.61.2 all worked and the data was migrated. From 0.61.2 to 0.62.2, the same crash occurs:

Here's the only log:

13.3.2018 06:49:09Will load cache for users
13.3.2018 06:49:0929 records load from users
13.3.2018 06:49:09Will load cache for rocketchat_room
13.3.2018 06:49:09179 records load from rocketchat_room
13.3.2018 06:49:09Will load cache for rocketchat_subscription
13.3.2018 06:49:09435 records load from rocketchat_subscription
13.3.2018 06:49:09Will load cache for rocketchat_settings
13.3.2018 06:49:09614 records load from rocketchat_settings
13.3.2018 06:49:12Updating process.env.MAIL_URL
13.3.2018 06:49:12Starting Email Intercepter...

Server Setup Information:

Steps to Reproduce:

See description

Expected behavior:

Upgrade should succeed

Actual behavior:

Rocket.Chat terminates.

Relevant logs:

See above.

ghost commented 6 years ago

Nodejs Version is pretty important right now. "dpkg -l | grep nodejs" will give you that information.

micw commented 6 years ago

docker run --rm -it library/rocket.chat:0.62.2 node --version v8.10.0

(I'm using the official image, added the information above)

ghost commented 6 years ago

Well then you are out of luck. You have to downgrade your nodejs version.

https://github.com/nodejs/node/issues/19274

micw commented 6 years ago

Thank you a lot. That means the official Rocket.Chat docker image is broken. I'll file a bug report in that repo.

sampaiodiego commented 6 years ago

@micw the logs you showed does not contains any errors.. how long did you wait for it to start?

I've tested the official image and seems it is working. here are the full logs I got (notice Node version):

$ docker run --rm -it --link mongo -e MONGO_URL=mongodb://mongo/dockerofficial -e MONGO_OPLOG_URL=mongodb://mongo/local rocket.chat:0.62.2
Will load cache for users
0 records load from users
Will load cache for rocketchat_room
0 records load from rocketchat_room
Will load cache for rocketchat_subscription
0 records load from rocketchat_subscription
Will load cache for rocketchat_settings
0 records load from rocketchat_settings
Updating process.env.MAIL_URL
Starting Email Intercepter...
Will load cache for rocketchat_permissions
0 records load from rocketchat_permissions
Will load cache for rocketchat_roles
0 records load from rocketchat_roles
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
Setting default file store to GridFS
LocalStore: store created at
LocalStore: store created at
connect deprecated multipart: use parser (multiparty, busboy, formidable) npm module instead npm/node_modules/connect/lib/middleware/bodyParser.js:56:20
connect deprecated limit: Restrict request size at location of read npm/node_modules/connect/lib/middleware/multipart.js:86:15
Using GridFS for custom sounds storage
Using GridFS for custom emoji storage
ufs: temp directory created at "/tmp/ufs"
Updating process.env.MAIL_URL
➔ System ➔ startup
➔ +----------------------------------------------+
➔ |                SERVER RUNNING                |
➔ +----------------------------------------------+
➔ |                                              |
➔ |  Rocket.Chat Version: 0.62.2                 |
➔ |       NodeJS Version: 8.10.0 - x64           |
➔ |             Platform: linux                  |
➔ |         Process Port: 3000                   |
➔ |             Site URL: http://localhost:3000  |
➔ |     ReplicaSet OpLog: Enabled                |
➔ |          Commit Hash: 07f0e3f77c             |
➔ |        Commit Branch: HEAD                   |
➔ |                                              |
➔ +----------------------------------------------+
micw commented 6 years ago

@sampaiodiego The container dies immediately after the log line I copied. Maybe it's an issue with the existing data? As you can see, my installation reads some data while yours reads 0 records. Any chance to enable some debug log for the docker version?

bbrauns commented 6 years ago

hey there, I have a similar issue with docker image version rocket.chat:0.62.2 but not with rocket.chat:0.62.1. I saw that 0.62.1 uses Node v8.9.4 and 0.62.2 uses Node v8.10.0.

Before the container crashes I see the following message in syslog: Mar 14 09:25:45 chat-stage kernel: [372226.247076] traps: node[61380] general protection ip:d8b6b7d5408 sp:7f8878702a18 error:0.

Anyone knows how to further debug this issue?

sampaiodiego commented 6 years ago

@bbrauns I've created a pull request https://github.com/RocketChat/Docker.Official.Image/pull/42 so official docker image will use 8.9.4 too..

in the meantime you can use our own image rocketchat/rocket.chat:0.62.2 which is using node 8.9 and is working for @micw

gcommit commented 6 years ago

I have the same problem. It repeats over and over again the message

Will load cache for users 2 records load from users Will load cache for rocketchat_room 1 records load from rocketchat_room Will load cache for rocketchat_subscription 0 records load from rocketchat_subscription Will load cache for rocketchat_settings 610 records load from rocketchat_settings Updating process.env.MAIL_URL Starting Email Intercepter...

But it does not start at all...

Same issue with 0.62.1, only 0.62.0 is starting up on my side, but there i have this issue: https://github.com/RocketChat/Rocket.Chat/issues/10136

sampaiodiego commented 6 years ago

do you guys have enabled oplog (have replica set enabled on MongoDB and are setting the env var MONGO_OPLOG_URL to start up rocket.chat)?

I was able to reproduce the error using official Docker image without oplog enabled. it works fine with oplog enabled though (which is the recommended the setup)

micw commented 6 years ago

I have no oplog enabled and run a single node mongodb

sampaiodiego commented 6 years ago

@micw even on single node mongodb setup using replica set is recommended. you may have issues as your rocket.chat grow up

TomaszDom commented 6 years ago

FWIW, I'm running 0.62.2 dual instance with oplog enabled and latest node lts version (8.10.0) with no issues.

bbrauns commented 6 years ago

@sampaiodiego oplog has always been activated:

root@chat-stage:~/rocket-chat# cat docker-compose.yml
services:
  app:
    depends_on:
    - mongo
    environment:
    - PORT=3000
    - ROOT_URL=xyz
    - MONGO_URL=mongodb://mongo:27017/rocketchat
    - MONGO_OPLOG_URL=mongodb://mongo:27017/local
    image: rocket.chat:0.62.2
    restart: always
    volumes:
    - /data/upload:/data/upload
  haproxy:
    container_name: haproxy
    hostname: haproxy
    image: haproxy:1.7
    ports:
    - 80:8080
    - 81:8081
    - 82:8082
    - 443:8443
    restart: always
    volumes:
    - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    - ./chat-combined.pem:/usr/local/etc/haproxy/chat-combined.pem:ro
    - /dev/log:/dev/log:rw
  mongo:
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    image: mongo:3.4
    logging:
      driver: syslog
      options:
        tag: '{{.Name}}'
    ports:
    - 127.0.0.1:27017:27017
    restart: always
    volumes:
    - /data/db:/data/db
  mongo-init-replica:
    command: 'mongo mongo/rocketchat --eval "rs.initiate({ _id: ''rs0'', members:
      [ { _id: 0, host: ''localhost:27017'' } ]})"'
    depends_on:
    - mongo
    image: mongo:3.4
version: '2'
brtptrs commented 6 years ago

We had a similar problem which seems to have disappeared after downgrading to node 8.9.4. We hat the restart problem on a clean install of version 0.62.1

pauloriply commented 6 years ago

Same problem with installed rocket.chat via snap... I have no idea what can be done to raise the server UPD: in my case help dist-upgrade... I do not know what happened, but the chat started normaly Ubuntu 16.04

AleksanderGrzybowski commented 6 years ago

Same here, I wanted to try this out, but it crashes on the first restart, for now I'll pick older docker image, but would really like to see it fixed :)

TwizzyDizzy commented 6 years ago

As far as I see it, this (#10331) is the issue we need to keep track of, isn't it? Docker files have been downgraded to my knowledge (and will get upgraded once #10331 is done - which is the case when a newer nodejs version is released which fixes the SEGVs).

So this issue is redundant, isn't it? If not, please let me know :)

Cheers Thomas

micw commented 6 years ago

@TwizzyDizzy I'm not exactly sure if it's the very same issue. While in #10331 users complain about random segfaults, this issue is about a reproducable crash on startup. Nevertheless, both seems to be related to the nodejs version.

TwizzyDizzy commented 6 years ago

Mh. Well then I'm just gonna check back here when the newer nodejs version is released.

Cheers Thomas

TwizzyDizzy commented 6 years ago

New nodejs-Version has been released. Please give me some feedback whether the issue at hand is still relevant. Otherwise I'm gonna close it in the near future.

Cheers Thomas

TwizzyDizzy commented 6 years ago

@rocket-cat close

Closing due to lack of feedback. If this is still an issue, please get back to me and I'll consider reopening based on the given information.

Cheers Thomas