Closed francisco-core closed 4 years ago
I think the relevant part might be:
M 20 Jan 2020 16:02:40.093 # Could not create server TCP listening socket *:6379: bind: Address already in use
signal-handler (1579536190) Received SIGTERM scheduling shutdown...
But it's too verbose for me to parse without a headache. What shall we do?
I was trying to run the discourse role to update the forum but it seems to fail. The problem seems unrelated with the change of version as running with the current version yields a similar failure state
Did you update the Discourse version?
The version in master (https://github.com/PrivacyLx/devops/blob/master/ansible/roles/discourse/defaults/main.yml#L10) is v2.3.7
.
It might be better from now on to test these updates
It's always better to test the updates and preferably let the people know that you are trying to upgrade the forum so that other will know that the forum is down for maintenance and not unreachable for whatever reason.
But it's too verbose for me to parse without a headache. What shall we do?
It seems that log output in your previous comment (https://github.com/PrivacyLx/privacylx-issue-tracker/issues/62#issue-552388783) is not complete.
Did you try to run the ansible role on a test server?
I think I had mentioned to you before that I do not have access to a test server with enough ram to run this. And last time I did the upgrade it went smoothly.
It seems that log output in your previous comment (#62 (comment)) is not complete.
Damn. I've already lost the log now. Do you want me to crash it again?
It's always better to test the updates and preferably let the people know that you are trying to upgrade the forum so that other will know that the forum is down for maintenance and not unreachable for whatever reason.
I should have done that. Next time, I'll add a banner at the top one day prior to announce it.
I ran it in a VM now with v2.3.7
and this is the final part of the log (without ansible)
HEAD is now at 5b17584adb Version bump to v2.3.7
I, [2020-01-21T12:19:49.001637 #1] INFO -- :
I, [2020-01-21T12:19:49.001740 #1] INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2020-01-21T12:19:49.004210 #1] INFO -- :
I, [2020-01-21T12:19:49.004298 #1] INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2020-01-21T12:19:49.006426 #1] INFO -- :
I, [2020-01-21T12:19:49.006508 #1] INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2020-01-21T12:19:49.008455 #1] INFO -- :
I, [2020-01-21T12:19:49.008616 #1] INFO -- : > cd /var/www/discourse && mkdir -p /shared/log/rails
I, [2020-01-21T12:19:49.011116 #1] INFO -- :
I, [2020-01-21T12:19:49.011226 #1] INFO -- : > cd /var/www/discourse && bash -c "touch -a /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2020-01-21T12:19:49.014986 #1] INFO -- :
I, [2020-01-21T12:19:49.015181 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2020-01-21T12:19:49.019020 #1] INFO -- :
I, [2020-01-21T12:19:49.019153 #1] INFO -- : > cd /var/www/discourse && bash -c "mkdir -p /shared/{uploads,backups}"
I, [2020-01-21T12:19:49.023382 #1] INFO -- :
I, [2020-01-21T12:19:49.023528 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/{uploads,backups} /var/www/discourse/public"
I, [2020-01-21T12:19:49.027859 #1] INFO -- :
I, [2020-01-21T12:19:49.028084 #1] INFO -- : > cd /var/www/discourse && bash -c "mkdir -p /shared/tmp/{backups,restores}"
I, [2020-01-21T12:19:49.033015 #1] INFO -- :
I, [2020-01-21T12:19:49.033158 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2020-01-21T12:19:49.037877 #1] INFO -- :
I, [2020-01-21T12:19:49.038148 #1] INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2020-01-21T12:19:49.042099 #1] INFO -- :
I, [2020-01-21T12:19:49.042443 #1] INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2020-01-21T12:19:49.042883 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2020-01-21T12:19:50.982541 #1] INFO -- :
I, [2020-01-21T12:19:50.982845 #1] INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2020-01-21T12:19:50.985023 #1] INFO -- :
I, [2020-01-21T12:19:50.985211 #1] INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2020-01-21T12:19:50.986801 #1] INFO -- :
I, [2020-01-21T12:19:50.986915 #1] INFO -- : > mkdir -p /var/nginx/cache
I, [2020-01-21T12:19:50.988724 #1] INFO -- :
I, [2020-01-21T12:19:50.988949 #1] INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2020-01-21T12:19:50.989231 #1] INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2020-01-21T12:19:50.989440 #1] INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2020-01-21T12:19:50.989634 #1] INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2020-01-21T12:19:50.989821 #1] INFO -- : > echo "done configuring web"
I, [2020-01-21T12:19:50.991310 #1] INFO -- : done configuring web
I, [2020-01-21T12:19:50.991527 #1] INFO -- : > cd /var/www/discourse && gem update bundler
I, [2020-01-21T12:19:55.744394 #1] INFO -- : Updating installed gems
Updating bundler
Successfully installed bundler-2.1.4
Gems updated: bundler
I, [2020-01-21T12:19:55.744614 #1] INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
chown: cannot dereference '/var/www/discourse/public/plugins/discourse-internet-explorer': No such file or directory
I, [2020-01-21T12:19:56.418621 #1] INFO -- :
I, [2020-01-21T12:19:56.418955 #1] INFO -- : Terminating async processes
I, [2020-01-21T12:19:56.418982 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main pid: 50
I, [2020-01-21T12:19:56.419025 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 167
2020-01-21 12:19:56.419 UTC [50] LOG: received fast shutdown request
167:signal-handler (1579609196) Received SIGTERM scheduling shutdown...
2020-01-21 12:19:56.447 UTC [50] LOG: aborting any active transactions
2020-01-21 12:19:56.449 UTC [50] LOG: worker process: logical replication launcher (PID 59) exited with exit code 1
2020-01-21 12:19:56.449 UTC [54] LOG: shutting down
167:M 21 Jan 2020 12:19:56.516 # User requested shutdown...
167:M 21 Jan 2020 12:19:56.516 * Saving the final RDB snapshot before exiting.
167:M 21 Jan 2020 12:19:56.527 * DB saved on disk
167:M 21 Jan 2020 12:19:56.527 # Redis is now ready to exit, bye bye...
2020-01-21 12:19:56.536 UTC [50] LOG: database system is shut down
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+ failed with return #<Process::Status: pid 335 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"web", "cmd"=>["gem update bundler", "find $home ! -user discourse -exec chown discourse {} \\+"]}
7bc61ea3f7bb28ecf36bc23d1c942033a74f4c213d1f2084b758c6d7c90a0736
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
It seems to be failing on installing the default plugin discourse-internet-explorer
The bug is listed in: https://meta.discourse.org/t/error-rebuilding-on-2-3-8/136992/5
There is a fix but it is not included in a release yet: https://github.com/discourse/discourse/releases
I think I had mentioned to you before that I do not have access to a test server with enough ram to run this. And last time I did the upgrade it went smoothly.
Short term human memory may not be sufficient for everything ;) I opened an issue about it here: https://github.com/PrivacyLx/privacylx-issue-tracker/issues/63
Short term human memory may not be sufficient for everything ;) I opened an issue about it here: #63
Ofc! I don't blame you for that. Thanks
The bug is listed in: https://meta.discourse.org/t/error-rebuilding-on-2-3-8/136992/5
There is a fix but it is not included in a release yet: https://github.com/discourse/discourse/releases
Well spotted! I had not yet come across that.
I think this update can wait, then since there is no known CVE for that v2.3.7
Update: the actual issue is that MaxMind DB cannot longer be used with its new license re: https://meta.discourse.org/t/upgrade-rebuilds-fail-due-to-maxmind-db-eol/137387
It will be fixed in the next upcoming release.
I was trying to run the discourse role to update the forum but it seems to fail. The problem seems unrelated with the change of version as running with the current version yields a similar failure state
The forum is now up and running again because I ran manually
/var/discourse/discourse-doctor
on the server.It might be better from now on to test these updates