search

RocketChat / install.sh

command line tool to help you install and configure a RocketChat server in a Linux host
https://rocket.chat/docs/installation/rocketchatctl/
32 stars 33 forks source link

Upgrade 3.7.1 → 3.9.0 fails #33

Closed m4z closed 3 years ago

m4z commented 3 years ago

When using rocketchatctl to update just now, I ran into https://github.com/RocketChat/Rocket.Chat/issues/19537. From the phrasing there I'd assume it would only happen on manual upgrade.

This is on Debian 9.13.

What I did

trolls:~ # rocketchatctl check-updates
Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.
Current update available for RocketChat server: from 3.7.1 to 3.9.0.

We need to patch rocketchatctl a bit, so I always do this:

trolls:~ # cp /usr/local/bin/rocketchatctl{,.$(date +%F)}

trolls:~ # rocketchatctl upgrade-rocketchatctl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40074  100 40074    0     0  56840      0 --:--:-- --:--:-- --:--:-- 56842
rocketchatctl upgraded to latest version.

trolls:~ # diff /usr/local/bin/rocketchatctl.2020-12-01 /usr/local/bin/rocketchatctl
44c44
<   --dir=<path_to_dir>       sets the directory for storing the backup files, default backup directory is /tmp
---
>   --backup-dir=<path_to_dir>       sets the directory for storing the backup files, default backup directory is /tmp
760c760
<         current_rocketchat_version=$(curl --interface lo http://localhost:$PORT/api/info 2>/dev/null |cut -d\" -f4)
---
>         current_rocketchat_version=$(curl http://localhost:$PORT/api/info 2>/dev/null |cut -d\" -f4)
1118c1118
<             --dir)
---
>             --backup-dir)

trolls:~ # vimdiff /usr/local/bin/rocketchatctl.2020-12-01 /usr/local/bin/rocketchatctl
2 files to edit

(Patched our "--interface lo" workaround back into it, otherwise we (IIRC) run into a problem because this is an IPv6 host and "localhost" resolves to "::1" first, but parts of Rocket.Chat run on "127.0.0.1" only.)

trolls:~ # rocketchatctl check-updates
Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.
Current update available for RocketChat server: from 3.7.1 to 3.9.0.

trolls:~ # mkdir /opt/$(date +%F)_backup-rocket.chat-db && rocketchatctl backup --backup-dir=/opt/$(date +%F)_backup-rocket.chat-db
2020-12-01T08:16:06.715+0000    writing rocketchat.view_livechat_queue_status to archive on stdout
[…]
2020-12-01T08:16:09.437+0000    done dumping rocketchat.rocketchat_statistics (7369 documents)
Backup done successfully, check for the backup file in /opt/2020-12-01_backup-rocket.chat-db/dump_1606810566.gz. You can restore using # mongorestore --host localhost:27017 --db rocketchat --archive --gzip < /opt/2020-12-01_backup-rocket.chat-db/dump_1606810566.gz

trolls:~ # rocketchatctl update
Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.
/usr/local/bin/rocketchatctl: line 900: [: 3.9.0: unary operator expected
Could not determine if updates available for RocketChat server.

(We regularly see this, and just re-run the command until it works.)

trolls:~ # rocketchatctl update
Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.
/usr/local/bin/rocketchatctl: line 900: [: 3.9.0: unary operator expected
Could not determine if updates available for RocketChat server.
trolls:~ # bash -x rocketchatctl update
+ set -o nounset
+ main update
+ local VERSION=latest
+ local -r NODE_VERSION=v12.14.0
+ local -r NPM_VERSION=6.13.4
+ local MONGO_VERSION=
+ local -r 'MONGO_URL=mongodb://localhost:27017/rocketchat?replicaSet=rs01'
+ local -r 'MONGO_OPLOG_URL=mongodb://localhost:27017/local?replicaSet=rs01'
+ local ROCKETCHAT_DOWNLOAD_URL=https://releases.rocket.chat/latest/download
+ local -r NODE_DEB_DOWNLOAD_URL=https://deb.nodesource.com/setup_12.x
+ local -r NODE_RPM_DOWNLOAD_URL=https://rpm.nodesource.com/setup_12.x
+ local -r ROCKETCHAT_DIR=/opt/Rocket.Chat
+ local -r ROCKETCHAT_DIR_UPDATE=/opt/Rocket.Chat.bkup
+ local -r ROCKET_VERSION_INFO_URL=https://releases.rocket.chat/latest/info
+ local -r ROCKETCHATCTL_DOWNLOAD_URL=https://raw.githubusercontent.com/RocketChat/install.sh/master/rocketchatctl
+ local -r ROCKETCHATCTL_DIRECTORY=/usr/local/bin
+ local -r ROCKETCHATCTL_TEMP_DIRECTORY=/tmp
+ local -r TRAEFIK_DOWNLOAD_URL=https://github.com/containous/traefik/releases/download/v1.7.20/traefik
+ local -r CADDY_DOWNLOAD_URL=https://github.com/mholt/caddy/releases/download/v1.0.0/caddy_v1.0.0_linux_amd64.tar.gz
+ local distro=
+ local distro_version=
+ local install_node=0
+ local install_mongo=0
+ local rocket_healthy=0
+ local ROOT_URL=
+ local PORT=3000
+ local BIND_IP=0.0.0.0
+ local bind_loopback=true
+ local webserver=none
+ local install_node_arg=0
+ local use_mongo_arg=0
+ local webserver_arg=0
+ local rocket_mail_arg=0
+ local bind_loopback_arg=0
+ local rocket_domain=
+ local configure_rocketchat=0
+ local configure_webserver=0
+ local root_url_arg=0
++ date +%s
+ local sufix=1606810639
+ local node_path=/usr/bin/node
+ local backup_dir=/tmp
+ local update_node=0
+ '[' 0 -ne 0 ']'
+ '[' 1 -lt 1 ']'
+ [[ -t 0 ]]
+ case $1 in
+ os_supported
+ get_os_distro
+ '[' -r /etc/os-release ']'
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 9 (stretch)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=9
+++ VERSION='9 (stretch)'
+++ VERSION_CODENAME=stretch
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ echo debian
+ distro=debian
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 9 (stretch)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=9
+++ VERSION='9 (stretch)'
+++ VERSION_CODENAME=stretch
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ echo 9
+ distro_version=9
+ case "$distro" in
+ [[ 9 =~ ((9|10)) ]]
+ do_check_updates
+ get_rocketchat_latest_version
+ command_exists curl
+ command -v curl
++ curl https://releases.rocket.chat/latest/info
++ grep -w tag
++ awk '-F"' '{print $2}'
++ awk -F: '{print $2}'
+ latest_rocketchat_version=3.9.0
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ cat /lib/systemd/system/rocketchat.service
++ grep PORT
++ awk -F= '{print $3}'
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=3.7.1
+ print_make_backup
+ echo 'Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.'
Updates could be risky, you can use the backup option # rocketchatctl backup, to create a backup of the rocketchat database first. Please check that you have enough space in your system to store the backup.
+ [[ 3.7.1 =~ ^([0-2]\..\..)$ ]]
+ version_gt 3.9.0 3.7.1
++ printf '%s\n' 3.9.0 3.7.1
++ sort -V
++ head -n 1
+ test 3.7.1 '!=' 3.9.0
+ print_update_available_and_exit
+ echo 'Current update available for RocketChat server: from 3.7.1 to 3.9.0.'
Current update available for RocketChat server: from 3.7.1 to 3.9.0.
+ do_update
+ '[' 0 -eq 1 ']'
+ mv /opt/Rocket.Chat /opt/Rocket.Chat.bkup
+ '[' 0 == 0 ']'
+ systemctl stop rocketchat
+ install_rocketchat
+ curl -L https://releases.rocket.chat/latest/download -o /tmp/rocket.chat.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   132  100   132    0     0    264      0 --:--:-- --:--:-- --:--:--   264
100  148M  100  148M    0     0   447k      0  0:05:40  0:05:40 --:--:-- 30.2M
+ tar -xzf /tmp/rocket.chat.tgz -C /tmp
+ cd /tmp/bundle/programs/server
+ npm install
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported

> fibers@4.0.3 install /tmp/bundle/programs/server/node_modules/fibers
> node build.js || nodejs build.js

`linux-x64-72-glibc` exists; testing
Binary is fine; exiting
npm WARN lifecycle meteor-dev-bundle@~install: cannot run in wd meteor-dev-bundle@ node npm-rebuild.js (wd=/tmp/bundle/programs/server)
added 146 packages from 122 contributors and audited 147 packages in 7.125s

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

+ mv /tmp/bundle /opt/Rocket.Chat
+ rm /tmp/rocket.chat.tgz
+ set_rocketchat_user_and_permissions
+ id -un rocketchat
+ '[' 0 -ne 0 ']'
+ chown -R rocketchat:rocketchat /opt/Rocket.Chat
+ systemctl start rocketchat
+ rocketchat_health_check
+ local count=0
+ '[' 0 -eq 1 ']'
+ '[' 0 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ cat /lib/systemd/system/rocketchat.service
++ grep PORT
++ awk -F= '{print $3}'
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=
+ [[ '' =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ sleep 2
+ (( count++ ))
+ echo 'Waiting up to 5 minutes for RocketChat server to be active ... 2'
Waiting up to 5 minutes for RocketChat server to be active ... 2
+ '[' 0 -eq 1 ']'
+ '[' 1 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ cat /lib/systemd/system/rocketchat.service
++ grep PORT
++ awk -F= '{print $3}'
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=
+ [[ '' =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ sleep 2
+ (( count++ ))
+ echo 'Waiting up to 5 minutes for RocketChat server to be active ... 4'
Waiting up to 5 minutes for RocketChat server to be active ... 4
+ '[' 0 -eq 1 ']'
+ '[' 2 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ cat /lib/systemd/system/rocketchat.service
++ awk -F= '{print $3}'
++ grep PORT
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=
+ [[ '' =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ sleep 2
+ (( count++ ))
+ echo 'Waiting up to 5 minutes for RocketChat server to be active ... 6'
Waiting up to 5 minutes for RocketChat server to be active ... 6
+ '[' 0 -eq 1 ']'
+ '[' 3 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ awk -F= '{print $3}'
++ grep PORT
++ cat /lib/systemd/system/rocketchat.service
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=
+ [[ '' =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ sleep 2
+ (( count++ ))
+ echo 'Waiting up to 5 minutes for RocketChat server to be active ... 8'
Waiting up to 5 minutes for RocketChat server to be active ... 8
+ '[' 0 -eq 1 ']'
+ '[' 4 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
++ awk -F= '{print $3}'
++ grep PORT
++ cat /lib/systemd/system/rocketchat.service
+ PORT=3000
++ curl --interface lo http://localhost:3000/api/info
++ cut '-d"' -f4
+ current_rocketchat_version=
+ [[ '' =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ sleep 2
+ (( count++ ))
+ echo 'Waiting up to 5 minutes for RocketChat server to be active ... 10'
Waiting up to 5 minutes for RocketChat server to be active ... 10
+ '[' 0 -eq 1 ']'
+ '[' 5 -eq 150 ']'
+ get_rocketchat_current_version
+ systemctl status rocketchat
+ print_rocketchat_not_running_error_and_exit
+ errcho 'RocketChat server not running. Cancelling'
+ echo 'RocketChat server not running. Cancelling'
RocketChat server not running. Cancelling
+ exit 1

(The download takes that long because, well, corporate proxy.)

syslog revealed this error:

Dec 01 08:23:25 trolls systemd[1]: Started The Rocket.Chat server.
Dec 01 08:23:35 trolls rocketchat[17837]: /opt/Rocket.Chat/programs/server/node_modules/fibers/future.js:280
Dec 01 08:23:35 trolls rocketchat[17837]:                                                 throw(ex);
Dec 01 08:23:35 trolls rocketchat[17837]:                                                 ^
Dec 01 08:23:35 trolls rocketchat[17837]: TypeError: AsyncLocalStorage is not a constructor
Dec 01 08:23:35 trolls rocketchat[17837]:     at module (server/sdk/index.ts:23:34)
Dec 01 08:23:35 trolls rocketchat[17837]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Dec 01 08:23:35 trolls rocketchat[17837]:     at Module.require (packages/modules-runtime.js:238:14)
Dec 01 08:23:35 trolls rocketchat[17837]:     at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/reify/lib/runtime/index.js:52:22)
Dec 01 08:23:35 trolls rocketchat[17837]:     at ee/server/broker.ts:1:80
Dec 01 08:23:35 trolls rocketchat[17837]:     at module (ee/server/broker.ts:345:2)
Dec 01 08:23:35 trolls rocketchat[17837]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Dec 01 08:23:35 trolls rocketchat[17837]:     at Module.require (packages/modules-runtime.js:238:14)
Dec 01 08:23:35 trolls rocketchat[17837]:     at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/reify/lib/runtime/index.js:52:22)
Dec 01 08:23:35 trolls rocketchat[17837]:     at module (server/main.js:1:8)
Dec 01 08:23:35 trolls rocketchat[17837]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Dec 01 08:23:35 trolls rocketchat[17837]:     at Module.require (packages/modules-runtime.js:238:14)
Dec 01 08:23:35 trolls rocketchat[17837]:     at require (packages/modules-runtime.js:258:21)
Dec 01 08:23:35 trolls rocketchat[17837]:     at /opt/Rocket.Chat/programs/server/app/app.js:191196:15
Dec 01 08:23:35 trolls rocketchat[17837]:     at /opt/Rocket.Chat/programs/server/boot.js:401:38
Dec 01 08:23:35 trolls rocketchat[17837]:     at Array.forEach (<anonymous>)
Dec 01 08:23:35 trolls systemd[1]: rocketchat.service: Main process exited, code=exited, status=1/FAILURE
Dec 01 08:23:35 trolls systemd[1]: rocketchat.service: Unit entered failed state.
Dec 01 08:23:35 trolls audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rocketchat comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Dec 01 08:23:35 trolls systemd[1]: rocketchat.service: Failed with result 'exit-code'.

That in turn led me to https://github.com/RocketChat/Rocket.Chat/issues/19537.

Workaround

This workaround (distilled from the above issue, and augmented) helped:

Before:

# node --version
v12.14.0

Workaround:

# n install 12.18.4
# systemctl restart rocketchat.service

HTH