DMS - A proposal and implementation of a design change to address many open issues

[schaefi]

Hi, I was looking at the DMS, the open bugs, the ongoing reports about non-working migrations, the big number of limitations in the current design and the lack of knowledge on the support side to allow proper debugging when issues occurs. All this was the driver to suggest a design change and an implementation of how I see a maintainable future of the DMS. If you would like to review the changes please do so based on the list of commits, they also fix some bugs in the implementation which could be merged independently of the design change.

I marked the PR with the workshop label as I'd like to have a conversation/demo on the topic to gather feedback and if the proposed change is considered an acceptable solution. There are presentation slides to provide a high level description of the proposal and I will attach them as follows:

Distribution Migration System

The slide deck also shows a list of defects that I believe will be solved if we go that route.

Long story short if you would like to test an upgrade from SLE12 to SLE15-SP5 which is not possible with the current DMS you can do so in the changed implementation from here as follows:

1. Start a SLE12 (x86_64) OnDemand instance in some cloud and ssh into it. From there:

sudo zypper ar https://download.opensuse.org/repositories/home:/marcus.schaefer:/dms/SLE_12_SP5 Migration
sudo zypper install suse-migration

sudo migrate --reboot

... a few notes:

• The packages installed from my home: project should be made available in the SLE12 public cloud module for a final release such that adding that extra repo will not be needed anymore. We already have this concept in place.
• You can continue to use your instance while it upgrades
• The new migrate tool is the user facing entry-point. There won't be any reboot/kexec in the process except for the final reboot after upgrade to activate the new kernel and instance registration. This is controlled via the --reboot flag such that a user can determine the time for the reboot himself. However, I strongly recommend to immediately reboot, hence used as such in the above example

I believe there will be many questions. Let's discuss this during the workshop with the implementation at hand.

Tasks not done yet:

• [ ] Update documentation
• [ ] Plan package delivery

Thanks

[rjschwei]

As stated when the idea about the containerization came up, a container needs a place to run, i.e. it needs a running system. If that containers runs on the system to be upgraded that system is no longer off line and on line migration across major releases is not supported. As such a conversation on how this is expected to work is definitely necessary.

[schaefi]

As stated when the idea about the containerization came up, a container needs a place to run, i.e. it needs a running system. If that containers runs on the system to be upgraded that system is no longer off line and on line migration across major releases is not supported. As such a conversation on how this is expected to work is definitely necessary.

The simplest way to see how and that this works is to try out the three lines of example commands above on a SLE12 system. I have tested both the zypper migration plugin as well as the zypper dup operations.

If that containers runs on the system to be upgraded that system is no longer off line and on line migration across major releases is not supported.

I believe the word off-line is misinterpreted in this context. We both know that SUSE explicitly documents that any zypper operation no matter if it's through the migration plugin or by a direct zypper dup call is not supported on the SAME system to run a major release upgrade. The point here is SAME system. With the move of the process into a container of the migration target you are no longer on the same system and therefore fulfill the requirement named off-line

Of course I was concerned about the following aspects:

• is a shared mount consistent across the entire upgrade process ?

  The answer is yes. A shared mount of / into the container as /system-root is an overlay mount in RAM and in the same way consistent as the mount process that we have in the live ISO

• is a host network connection consistent across the entire upgrade process ?

  The answer is yes. Any network interface is configured in the kernel and holds its data and stack in RAM until it gets re-configured. The installation of packages are not expected to re-configure the network. If they do this it would be broken no matter if you run the process in a container or on a live ISO. In all my tests the network and also the bridge setup into the container stays persistent and active. As I said the instance stays responsive during the whole process, a big plus in my eyes

• is the functionality of the running container instance harmed by the install of a new kernel/modules ?

  The answer here is no. What you need from the kernel is an active network (network drivers and bridge) as well as an overlay mount. All this is either provided as modules already in RAM or compiled into the kernel and all present before you start the migration. So there is no feature on the running system which would kill the container instance by an update of a package. The container storage on the sle12 system lives in /var/lib/containers/storage/ and is also wiped at the end of a successful migration to get rid of any artifacts. The data on this storage location is not touched by a zypper based upgrade process, also not when podman or any other container related package gets upgraded. I did a very intrusive test to this case by deleting everything from system-root except /var/lib/containers/storage from inside of a running container instance attached to a shell. The deletion went through and killed the complete sle12 system but the container instance as well as my shell stayed alive ;)

So I knew there will be tough questions like this :) but I believe I have done quite some tests and wrapped my head around several cases. That doesn't mean that I could not have forgotten something important, but this is exactly the reason why I created this PR and also packaged everything up such that you can do experiments as you see fit. It doesn't take away much of your time and you can do it in 5 minutes on AWS

[rjschwei]

I believe the word off-line is misinterpreted in this context.

I do not understand how it can be misinterpreted, the documentation is very clear and explicit about this. From [1]

""" Upgrading offline implies that the operating system to be upgraded is not running (system down state). """

When the kernel of the system to be upgraded is running the system is not in a "system down" state. I do not see where there is room for interpretation.

We both know that SUSE explicitly documents that any zypper operation no matter if it's through the migration plugin or by a direct zypper dup call is not supported on the SAME system to run a major release upgrade. The point here is SAME system. With the move of the process into a container of the migration target you are no longer on the same system and therefore fulfill the requirement named off-line

No, the requirement for the migration across major distributions is "system down" for the system that is to be upgraded [1]. this requirement is also why the SUMa process for migration across major version still depends on AutoYaST [2], they also make sure that no part of the system to be upgraded is running. All migration tooling at SUSE is build around this "system down" requirement/assumption, spare micro where the update goes into a new snapshot and as such all of these shenanigans are avoided.

[1] https://documentation.suse.com/sles/15-SP5/html/SLES-all/cha-upgrade-paths.html#sec-upgrade-paths-supported [2] https://documentation.suse.com/suma/5.0/en/suse-manager/common-workflows/workflow-inplace-sles-upgrade.html

Of course I was concerned about the following aspects:

• is a shared mount consistent across the entire upgrade process ? The answer is yes. A shared mount of / into the container as /system-root is an overlay mount in RAM and in the same way consistent as the mount process that we have in the live ISO
• is a host network connection consistent across the entire upgrade process ? The answer is yes. Any network interface is configured in the kernel and holds its data and stack in RAM until it gets re-configured. The installation of packages are not expected to re-configure the network. If they do this it would be broken no matter if you run the process in a container or on a live ISO. In all my tests the network and also the bridge setup into the container stays persistent and active. As I said the instance stays responsive during the whole process, a big plus in my eyes
• is the functionality of the running container instance harmed by the install of a new kernel/modules ? The answer here is no. What you need from the kernel is an active network (network drivers and bridge) as well as an overlay mount. All this is either provided as modules already in RAM or compiled into the kernel and all present before you start the migration. So there is no feature on the running system which would kill the container instance by an update of a package. The container storage on the sle12 system lives in /var/lib/containers/storage/ and is also wiped at the end of a successful migration to get rid of any artifacts. The data on this storage location is not touched by a zypper based upgrade process, also not when podman or any other container related package gets upgraded. I did a very intrusive test to this case by deleting everything from system-root except /var/lib/containers/storage from inside of a running container instance attached to a shell. The deletion went through and killed the complete sle12 system but the container instance as well as my shell stayed alive ;)

So I knew there will be tough questions like this :) but I believe I have done quite some tests and wrapped my head around several cases. That doesn't mean that I could not have forgotten something important, but this is exactly the reason why I created this PR and also packaged everything up such that you can do experiments as you see fit. It doesn't take away much of your time and you can do it in 5 minutes on AWS

It is not about whether or not it functions. It is about what support do we get from the rest of the organization. If we create a system that does not meet the documented requirements, other teams, especially support, can state that the requirements for major distribution migration are not met. And they will be correct. That in turn means we sign up for the support of the whole stack instead of only supporting the mechanism that drives the migration stack. This is something we cannot do and we risk leading the customer into an unsupported scenario.

That said there are of course arguments to be made that what we have today might be too strict. That means we have to have conversations with technical leadership in BCL and we have to include PMs in the conversation as well. Although I think PM agreement on a scenario which promises to make major distribution migration simpler is a given.