Upgrade errors can result in situations where full restarts are needed

[Qqwy]

Steps to reproduce

I recently moved a module out to its own separate package. Once attempting to deploy, I got a Multiple defined modules error. Similar cases exist (e.g. 'wrong version of ERTS' etc) that result in a release breaking the hot-upgrade process halfway through.

However, this was impossible to fix without a non-hot upgrade, because hot upgrades are not atomic: After copying the release.tar.gz, Distillery unpacks it and adds info to some configuration files; removing the 'broken' release is virtually impossible, so one is forced to perform a non-hot upgrade instead.

Proposed fix

• Instead of saying 'release x.y.z is already unpacked', maybe check if there's a new .tar.gz and always prefer unpacking (and afterwards deleting the .tar.gz) over skipping it.
• Or, instead, on having a problem while deploying a release, remove it from the configuration files, such that deployments really become atomic.

[bitwalker]

To be clear, Distillery doesn't do the unpacking, the release handler does. Upgrades are atomic up to a certain point, which in the relup is called point_of_no_return, after which failures will require a reboot of the VM to address. It is unlikely that we can do anything to improve this, at least outside of OTP itself.

If you have some specific scenarios I can use to reproduce, we can definitely see if there are things Distillery can do to improve though.

As an aside, I'm not sure why you got the 'multiple defined modules' when applying the upgrade, Distillery prevents you from building releases with conflicting modules, but perhaps there is another reason why that occurred, if you have a way for me to reproduce, I'd like to investigate that.

A few things to keep in mind:

• If you made a mistake with an upgrade, and it failed before the point of no return, you can bump the version number with the fix needed and bypass the failed version
• If the release fails after the point of no return, it cannot be recovered with a hot upgrade, a full restart is required
• I would make a habit of always running a test environment where you can apply your upgrades and verify that they are successful, this way you can avoid taking down your production instances due to an issue with the package. All you need for a test environment is the version currently running in production, so you can even spin up such an environment on demand. Testing like this is a crucial part of deploying with hot upgrades, as you generally only use them in cases where you can't afford downtime, so you need to verify that the upgrade is applied successfully and that the system behaves as expected post-upgrade - you don't want to be doing that for the first time in production.

[Qqwy]

So in this case, I:

• First had the module PlangaPhoenix that was part of the main Planga/:planga application.
• Then, this module was split off to its own library planga_phoenix; (The commit from the main Planga repo is here).
• Then, when attempting to install this, the same module this was in the old :planga app, and in the new :planga_phoenix app, and that was probably why the error occured.

In this case, I 'fixed' it by:

• rolling back the change in my local source repository.
• Renaming the module to PlangaPhoenix2 for the time being.
• Upgrading to this new version.
• Nów changing the dependency to the external version (So now there was no conflict, since the old module was called PlangaPhoenix2 and the new one PlangaPhoenix).
• Creating another version that removes the PlangaPhoenix2-module from the project alltogether.

I did have to install these manually (rather than using a hot upgrade) because obviously the error occured after the 'point of no return' that @bitwalker mentioned.

The main problem that Distillery might be able help with is then, that when you bump the version number with the fix needed after you've built a version that failed, Distillery will create the .relup/.appup from the broken version to the new one, rather than from the last working version to the new one. (So if you have 1, 2-broken and are building 3, then Distillery's .relup/.appup will go from 2-broken to 3, whereas the application is still on 1 and therefore cannot get to 3.)

Actually, would there be a way to tell Distillery to create .relup/.appups for the last N versions, rather than only the last one? Because that would not only be useful in this scenario, but also in the scenarios where you deploy and find out that you have some sort of logical regression (the deployment succeeds but you application has unwanted behaviour) that have you roll back, because currently in those situations, you have to go through the 'bad' version to get to the new verison as well.

[bitwalker]

Distillery won't know that in the 1/2/3 scenario that 2 was bad, you can tell Distillery that though using the --upfrom=1 flag to mix release. You can use this to upgrade from any version V to version V+N.

In cases where your upgrade succeeds, but has a bug, you can downgrade to the last version, using bin/myapp downgrade <last version>. You can do this multiple times, effectively reversing the order in which upgrades were applied. The only restriction is that you can only downgrade to a version which has a downgrade path defined from the current version. So if you built an upgrade from 1 to 2, and from 2 to 3, you can downgrade from 3 to 2, but you can't downgrade from 3 to 1 directly, you have to go 3->2, then 2->1.

I think you probably are correct about why the duplicate module situation occurred, but I suspect it is specifically due to the fact that the same module was moved into a dependency, which would mean that planga_phoenix would get loaded before planga had a chance to be upgraded (and the old version of the module unloaded). This is a case where you'd want to perform an intermediate upgrade which either removed that module, or renamed it, and then do the upgrade which adds that module in the new library. It's certainly annoying, but one of the caveats of hot upgrades, in that you have to coordinate changes like these carefully to make sure they happen in a predictable fashion, not so unlike database migrations. I doubt there is much Distillery can do in this case, but I'll look into it, because even if we just had the opportunity to warn about it, that'd save a lot of time.

[Qqwy]

Note: I'd like to point out that recently with the change of :cowboy being required for Phoenix to just :plug_cowboy being required, the same issue ('duplicate modules') happened again.