Update alaveteli vers 0.38.0.0

[laurentS]

En vue du passage vers les fonctions pro, on doit commencer par une mise à jour vers la dernière version. La 0.38.0.0 devrait être dispo dans les jours qui viennent. Je continue ce ticket en anglais, vu qu'il est probable que d'autres sites alaveteli puissent y trouver un intérêt.

This ticket keeps track of our upgrade process from 0.33.0.0 to 0.38.0.0, in preparation for the move to PRO features.

[laurentS]

Upgrade to 0.34.0.0

• Trivial update by changing the alaveteli version number in the ansible script.
• Bug with our templates, which were all broken. Fixed by updating the base theme as described in the changelog
• Manually ran bundle exec rake temp:update_params_yaml by ssh'ing into the server
• Setup files backup using restic on top of the already existing postgresql backup, this is in preparation for the OS upgrade to 18.04 (as support for 16.04 is dropped from 0.35.0.0)

[laurentS]

Upgrade to ubuntu bionic 18.04

used do-release-upgrade as advised pretty much everywhere. The upgrade itself looks ok, but on production server, it breaks as pdftk is not available in bionic anymore, https://github.com/mysociety/alaveteli/issues/5025 and fix in https://github.com/mysociety/alaveteli/pull/5232/files#diff-d5eb0653888644638720a6b98d9d9f1d don't work anymore because cosmic is not supported anymore, so mirrors are down.

Fixed by installng pdftk from eoan instead of cosmic.

passenger setup is completely different for bionic vs xenial, so nginx wouldn't start either. Fixed by following https://www.phusionpassenger.com/docs/tutorials/deploy_to_production/installations/oss/ownserver/ruby/nginx/

opendkim refused to start on staging server because the default config was changed between xenial and bionic. Fixes are in the git log.

Just about everything else fell apart in various ways between the 2 servers, python, ruby, etc... I fixed most of them through manual trial and error.

Take away: If at all possible, install the newer OS on a fresh machine and move over to it instead of trying the ubuntu upgrade path which even for LTS versions is very much broken.

[laurentS]

upgrade to 0.35.0.0

Upgrade itself super easy, except our xapian index update seemed to have crashed mid way a week before. removing production.new and rerunning the update script fixed it.

geoip is a problem, the v1 database is not available anymore, so rails geoip:download_data does not run, and we're now unable to login to the site. Work around:

• create an account with maxmind at https://www.maxmind.com/en/geolite2/signup
• create a license key (free)
• download the Geolite2 Country database, and extract the *.mmdb file to the server, under <alaveteliroot>/vendor/data
• update config/general.yml to have GEOIP_DATABASE: vendor/data/GeoLite2-Country.mmdb
• restart nginx

This seems to be enough for the server to function normally.

[laurentS]

Upgrade to 0.36.0.0

The geoip data download is now part of the rails-post-deploy script, and fails for the same reason as above.

• run the CI job and let it fails during the rails-post-deploy script
• ssh into the server, edit the rails-deploy-before-down script to comment out the line related to geoip at the end of the file
• run sudo -u alaveteli script/rails-post-deploy manually
• restart nginx

[laurentS]

Upgrade to 0.37.0.0

Patched config/routes.rb in our ansible setup as we overwrite the original one to use translated routes.

[laurentS]

Upgrade to 0.38.0.0

• we're already on xapian 1.4.5 so we don't seem to need to move from flint, as it's not supported anymore

• we're on postgresql 9.6, so no need to upgrade this

• we're using ruby 2.5, which seems fine from the alaveteli release notes

• first run of the CI process timed out on bundle install, and the server went into a 100% cpu usage, losing ssh connectivity

• after reboot, manual run of bundle install took a while but completed

• next CI run failed during rails-post-deploy, which was fixed as per https://github.com/mysociety/alaveteli/issues/5844

[laurentS]

This seems to be working ok now, closing.