search

freedomofpress / securedrop

GitHub repository for the SecureDrop whistleblower platform. Do not submit tips here!
https://securedrop.org/
Other
3.61k stars 685 forks source link

[xenial] Test upgrade path: Ubuntu 14.04.5 → 14.04.5 with all updates → 16.04 #3965

Closed eloquence closed 5 years ago

eloquence commented 5 years ago

Our plan of record is to upgrade from Ubuntu 14.04 to Ubuntu 16.04 by first ensuring that the system is updated to all latest packages available for Ubuntu 14.04 (not just security updates).

This is because the package version state of a SecureDrop server partially depends on whether or not the administrator has run the Ansible playbook, which causes a full package update.

To ensure that this strategy is viable, we should test the following upgrade path:

  1. Install from a 14.04.5 base image.
  2. Run apt-get update && apt-get upgrade
  3. Run do-release-upgrade

Beyond looking for showstopper bugs, we will want to compare the final system state with an install from a 16.04 base image to ensure that this upgrade does not result in odd inconsistencies that could cause problems later. This research may generate more tickets.

Part of #3204.

kushaldas commented 5 years ago 
Building dependency tree          
Reading state information... Done
Building data structures... Done 

Updating repository information

Third party sources disabled 

Some third party entries in your sources.list were disabled. You can 
re-enable them after the upgrade with the 'software-properties' tool 
or your package manager. 

To continue please press [ENTER]

While running do-release-upgrade we will see the above notice, this is a standard procedure from the Operating System vendors (think: Red Hat) to make sure that all third party vendor repos are disabled. That way, the chances of breakage due to missing packages(or dependencies) are low.

Even in the case of Fedora, we ask folks disable all third party repositories.

kushaldas commented 5 years ago

The xml error which we saw in https://github.com/freedomofpress/securedrop/issues/3968#issuecomment-450864877 does not occur if we execute just do-release-upgrade command. This is interesting.

kushaldas commented 5 years ago

These are various User INPUT I had to give during the upgrade process.

Updating repository information

Third party sources disabled 

Some third party entries in your sources.list were disabled. You can 
re-enable them after the upgrade with the 'software-properties' tool 
or your package manager. 

To continue please press [ENTER]

...

Do you want to start the upgrade? 

12 installed packages are no longer supported by Canonical. You can 
still get support from the community. 

19 packages are going to be removed. 141 new packages are going to be 
installed. 494 packages are going to be upgraded. 

You have to download a total of 248 M. This download will take about 
4 minutes with your connection. 

Installing the upgrade can take several hours. Once the download has 
finished, the process cannot be canceled. 

 Continue [yN]  Details [d]

screenshot from 2019-01-10 17-02-49

screenshot from 2019-01-10 17-05-20

screenshot from 2019-01-10 17-32-48

screenshot from 2019-01-10 17-33-17


Configuration file '/etc/modprobe.d/blacklist.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** blacklist.conf (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/moduli'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** moduli (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** ssh_config (Y/I/N/O/D/Z) [default=N] ?
Setting up openssh-sftp-server (1:7.2p2-4ubuntu2.6) ...
Setting up openssh-server (1:7.2p2-4ubuntu2.6) ...
Installing new version of config file /etc/network/if-up.d/openssh-server ...

Configuration file '/etc/pam.d/sshd'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sshd (Y/I/N/O/D/Z) [default=N] ? 

Processing triggers for resolvconf (1.78ubuntu6) ...
Errors were encountered while processing:
 initramfs-tools
Error in function: 

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (374.4 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C): K
eloquence commented 5 years ago

Just a note that part of the task here is to do a system state comparison (installed packages, state of relevant config files, etc.) of the upgraded system with a fresh 16.04 install -- we want to make sure that upgrading from 14.04.5 to 14.04.5 with all updates to 16.04 doesn't result in inconsistencies that could cause problems with SecureDrop.

kushaldas commented 5 years ago

Packages in the default xenial app (based on libvirt_staging_xenial_app_staging):

Pakcages in upgraded xenial box from production app vm

Packages in the default xenial mon (based on libvirt_staging_xenial_mon_staging)

Pakcages in upgraded xenial box from production mon vm

Many of the upgraded package names came up as they are of new version and the version name is part of the package name.

The default xenial image which we use in molecule also has a lot of extra packages, for example, lxd or cloud-* packages. Also wondering why Xenial has both systemd and upstart :)

The upgraded app vm gets gcc and g++ compilers,

kushaldas commented 5 years ago

While trying to upgrade the mon-server from the tails vm, I got the following. As you can see, many of those ncurses screens are totally messed up.

First, sudo vim /etc/update-manager/release-upgrades and changed the value of Prompt to lts.

upgrade_value_to_lts

Next, run sudo do-release-upgrade.

running_do_release_upgrade

Then, it will ask to press Enter.

mon_press_first_enter


Reading package lists... Done    
Building dependency tree          
Reading state information... Done
Building data structures... Done 

Updating repository information

Third party sources disabled 

Some third party entries in your sources.list were disabled. You can 
re-enable them after the upgrade with the 'software-properties' tool 
or your package manager. 

To continue please press [ENTER]

And after some time, it will give the details about upgrade and tell to press y to continue.

mon_press_y_to_continue

12 installed packages are no longer supported by Canonical. You can 
still get support from the community. 

10 packages are going to be removed. 144 new packages are going to be 
installed. 434 packages are going to be upgraded. 

You have to download a total of 243 M. This download will take about 
52 seconds with your connection. 

Installing the upgrade can take several hours. Once the download has 
finished, the process cannot be canceled. 

 Continue [yN]  Details [d]

Next, it will ask about restarting the services, I chose yes by using the TAB.

Then, it will ask about language selector, I have selected <Ok> by using the TAB key and pressed Enter. (You can see the messed up UI).

mon_language_selector

Next, it will give information about postfix and you have to press TAB to select <Ok> and pressed Enter.

mon_email_outgoing

Then, the actual postfix configuration screen, I have selected no configuration required (default value), and pressed TAB to select <Ok> and pressed Enter.

mon_postfix_configuration

Next, the following input is required. I kept pressing Enter for the default input.

Configuration file '/etc/modprobe.d/blacklist.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** blacklist.conf (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/moduli'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** moduli (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** ssh_config (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/pam.d/sshd'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sshd (Y/I/N/O/D/Z) [default=N] ?

After this it will show a few grub related screens and we will have to verify what we see on the actual hardware based installations. I did select any and kept pressing Enter.

mon_grub mon_grub_2 mon_no_update_grub

The next screen is about removing extra packages, where I pressed N and then pressed Enter.

mon_remove_packages_no

Searching for obsolete software
Reading package lists... Done    
Building dependency tree          
Reading state information... Done
Building data structures... Done 
Building data structures... Done 

Remove obsolete packages? 

46 packages are going to be removed. 

 Continue [yN]  Details [d]N

Restart required 

To finish the upgrade, a restart is required. 
If you select 'y' the system will be restarted. 

Continue [yN]

Press N and then it will ask you to press x to end the process. The Mon server should be restarted with Xenial.

kushaldas commented 5 years ago

A new error I saw today:


Calculating the changes

Calculating the changes

Could not calculate the upgrade 

An unresolvable problem occurred while calculating the upgrade. 

This can be caused by: 
* Upgrading to a pre-release version of Ubuntu 
* Running the current pre-release version of Ubuntu 
* Unofficial software packages not provided by Ubuntu 

If none of this applies, then please report this bug using the 
command 'ubuntu-bug ubuntu-release-upgrader-core' in a terminal. 

Restoring original system state

Aborting
Reading package lists... Done    
Building dependency tree          
Reading state information... Done
Building data structures... Done 
=== Command terminated with exit status 1 (Tue Feb  5 12:02:04 2019) ===

Seeing this again and again on my staging vm. @conorsch @eloquence @heartsucker @emkll @redshiftzero

kushaldas commented 5 years ago

Good part is the above error is only happening on staging, no clue on why.

emkll commented 5 years ago

12 installed packages are no longer supported by Canonical. You can still get support from the community.

I suspect this means they moved the channel from main to universe. We can find out what these packages are in /var/log/dist-upgrade/main.log searching for demoted:

2019-02-05 17:02:02,252 DEBUG demoted: 'biosdevname gcc-4.8-base gcc-4.9-base libarchive-extract-perl liblog-message-simple-perl libmodule-pluggable-perl libpod-latex-perl libterm-ui-perl libtext-soundex-perl module-init-tools python-debian w3m'
emkll commented 5 years ago

I am still not sure why wifi-related packages are being installed in the upgrade scenario, per the list provided here: https://github.com/freedomofpress/securedrop/issues/3965#issuecomment-454782755

zenmonkeykstop commented 5 years ago

Ran through do-release-upgrade via Tails admin against Monitor Server on physical hardware (7-series NUC). Prompts were as follows:

Update completes successfully, can't reconnect after initial reboot (might just be impatience on my part), but after rebooting again I can connect via ssh mon.

eloquence commented 5 years ago

OK, just to recap next steps here based on the problems reported:

eloquence commented 5 years ago

Closing in favor of #4163 and #4164, which should be tackled as part of QA during this sprint. We'll open an issue for the error Kushal saw in staging if we encounter it again during testing.