Closed josevalim closed 5 years ago
Hi folks,
I have spent most of the week studying hot code upgrades and building prototypes and I have decided to not include them as part of Elixir Core for now (at least for Elixir v1.9). The rationale is that they are still very complex, error prone and opinionated in a way that whoever is performing them needs to be aware of many of the decisions taken. For example, the distillery guides on appups (which is a part of hot code upgrades) covers many of those topics.
We are not saying they will never be part of Elixir but not for now. This can be a good opportunity for the community to build on top of what Elixir provides.
I wrote an extensive section on the docs about hot code upgrades which I reproduce below for convenience that explains this rationale and guides users.
Erlang and Elixir are sometimes known for the capability of upgrading a node that is running in production without shutting down that node. However, this feature is not supported out of the box by Elixir releases.
The reason we don't provide hot code upgrades is because they are very complicated to perform in practice, as they require careful coding of your processes and applications as well as extensive testing. Given most teams can use other techniques that are language agnostic to upgrade their systems, such as Blue/Green deployments, Canary deployments, Rolling deployments, and others, hot upgrades are rarely a viable option. Let's understand why.
In a hot code upgrade, you want to update a node from version A to
version B. To do so, the first step is to write recipes for every application
that changed between those two releases, telling exactly how the application
changed between versions, those recipes are called .appup
files.
While some of the steps in building .appup
files can be automated,
not all of them can. Furthermore, each process in the application needs
to be explicitly coded with hot code upgrades in mind. Let's see an example.
Imagine your application has a counter process as a GenServer:
defmodule Counter do
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
end
def bump do
GenServer.call(__MODULE__, :bump)
end
## Callbacks
def init(:ok) do
{:ok, 0}
end
def handle_call(:bump, counter) do
{:reply, :ok, counter + 1}
end
end
You add this process as part of your supervision tree and ship version
0.1.0 of your system. Now let's imagine that on version 0.2.0 you added
two changes: instead of bump/0
, that always increments the counter by
one, you introduce bump/1
that passes the exact value to bump the
counter. You also change the state, because you want to store the maximum
bump value:
defmodule Counter do
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
end
def bump(by) do
GenServer.call(__MODULE__, {:bump, by})
end
## Callbacks
def init(:ok) do
{:ok, {0, 0}}
end
def handle_call({:bump, by}, {counter, max}) do
{:reply, :ok, {counter + by, max(max, by)}}
end
end
If you to perform a hot code upgrade in such application, it would
crash, because in the initial version the state was just a counter
but in the new version the state is a tuple. Furthermore, you changed
the format of the call
message from :bump
to {:bump, by}
and
the process may have both old and new messages temporarily mixed, so
we need to handle both. The final version would be:
defmodule Counter do
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
end
def bump(by) do
GenServer.call(__MODULE__, {:bump, by})
end
## Callbacks
def init(:ok) do
{:ok, {0, 0}}
end
def handle_call(:bump, {counter, max}) do
{:reply, :ok, {counter + 1, max(max, 1)}}
end
def handle_call({:bump, by}, {counter, max}) do
{:reply, :ok, {counter + by, max(max, by)}}
end
def code_change(_, counter, _) do
{:ok, {counter, 0}}
end
end
Now you can proceed to list this process in the .appup
file and
hot code upgrade it. This is one of the many steps one necessary
to perform hot code upgrades and it must be taken into account by
every process and application being upgraded in the system.
The .appup
cookbook
provides a good reference and more examples.
Once .appup
s are created, the next step is to create a .relup
file with all instructions necessary to update the release itself.
Erlang documentation does provide a chapter on
Creating and Upgrading a Target System.
Learn You Some Erlang has a chapter on hot code upgrades.
Overall, there are many steps, complexities and assumptions made
during hot code upgrades, which is ultimately why they are not
provided by Elixir out of the box. However, hot code upgrades can
still be achieved by teams who desire to implement those steps
on top of mix release
in their projects or as separate libraries.
Although not achieved out of the box now, I think it is important to not only support them 'soon' but also have an easy way to test (ExUnit addition?) to make sure a given application 'can' be hot upgraded safely and for this to become part of the standard templates. Even if the feature is not used by an end-project this still helps ensure that program has a sense of 'migrations' and code_change's defined properly to reason about the information.
As for a more Elixir'y style, what about a set of migrations that migrate from version to version that define code_change functions and things that appup/relup's need and thus can be generated from that Migration code (ala Ecto.Migration
style).
The main issue I think would happen if its not supported 'soon' in the mix release
lifespan is that then a lot of libraries won't support it properly, and thus end projects that do use the functionality will get unexpected breakages that they may not notice for a period of time in production, where trying to enforce a way to do it well 'soon', especially with a testing infrastructure for it (given a 'migration' file and so forth and pulling different releases from hex/git of a library as an example then try to upgrade it it while performing tests) will help ensure that libraries handle it properly as soon as possible, thus making the ecosystem more hardy to the design.
Those are very good points @OvermindDL1. I particularly like the ExUnit idea.
I think it reinforces the point that it needs a lot of work and exploration before it gets to fully be part of core. We can also do it in steps too. For example, we can include a way to define .appups
and test them before we support relups in core per se.
But I also think we need to consider that, even if those functionalities are in Elixir, some people simply won't define the .appups
, so tracking if appups are available or not is also necessary.
It is quite easy to automate a lot of the .appup
work (as Distillery does), but it is still the case that code_change
implementations are entirely on users, including for dependencies, but in theory it should be possible to identify all the places where the state structure of a process has changed without a corresponding code_change
implementation; with a combination of comparing whether a module has changed, and analyzing the AST of the changed module (there are a few obvious places in the source to check, and if in doubt, be pessimistic and notify the user they need to check that module manually).
There are a number of people that have used hot upgrades with Distillery, and aside from some issues that were bugs in Distillery, I think the foundation it provided was better than having to do it all by hand.
I don't think hot upgrades are important to have right now, but I do think that there is a path towards supporting them in a way that simply takes away a lot of the manual work one has to do. In the near term, tools which build on core can provide hot upgrades (i.e. Distillery would likely continue to provide that capability as an additional feature).
The last PRs are here: #8966 #8967 #8957. With them merged, this is effectively done! Beta testers welcome! :heart:
@josevalim this is amazing! Thanks for all the awesome work as usual. I tried it out and it worked great on my first try! One question I couldn't find the answer to in the docs, though, is how to package the release into a tarball. Distillery creates the tarball for us, but I can't seem to find one generated here. Is it correct to simply pack up the entire _build/prod/rel/foo
folder? Would it make sense for the core elixir releases to provide a tarball like distillery does?
Hi @jesseshieh! I was actually planning to reach out and ask you if anything needs to be done for releases to just work in Gigalixir.
You are correct that right now we don't pack it up. We do have a :steps
feature where you could easily add your own step that packs it up. My only concern about packing it up is OS support. Should we do tar.gz on Unix and zip on Windows? But generally speaking, packing up the whole _build/prod/rel/foo is enough. After packed, where does distillery put the tar/zip?
@josevalim no problem at all. I should be able to handle packing the tarball easily. I think distillery puts it at _build/prod/rel/foo/releases/0.0.1/foo.tar.gz
, but it should be totally fine if you decide to put it somewhere else or even to not create one at all. I also don't mind implementing it, if you decide how/if to do it. Let me know and I'll create an issue to track it.
As far as getting releases to just work on gigalixir, I think things look pretty good as-is! I haven't tested things exhaustively yet, but almost everything I've run into are just minor syntactic differences like
bin/foo start
instead of bin/app foreground
mix release demo
instead of mix release --name=demo
. bin/foo remote
instead of bin/foo remote_console
I'm not sure it would make sense to add aliases for the sake of "backward compatibility" with distillery and it's pretty easy for me to support the new commands.
One thing I did notice, though, is that custom vm.args are not supported yet. Gigalixir currently uses custom vm.args with distillery to set the cookie and node name, but I think I can easily just use the new RELEASE_COOKIE
and RELEASE_NODE
env vars instead.
Actually, if you like, I don't mind taking a stab at adding support for custom vm.args if nobody else is currently working on it. Let me know, and I'll create an issue to track it.
bin/foo start instead of bin/app foreground
Actually, releases should be started by bin/start
. bin/start
is also meant to be customizable by users, for instance, you can set environment variables there too!
but I think I can easily just use the new RELEASE_COOKIE and RELEASE_NODE env vars instead.
I think that would be easier, yes. :) But since the cookie is read from a file, you can change the file too. It should all end-up the same.
Actually, if you like, I don't mind taking a stab at adding support for custom vm.args if nobody else is currently working on it.
I have a PR for it here: https://github.com/elixir-lang/elixir/pull/8967 :D Feedback is very appreciated.
Thanks for all of the input so far. Regarding the tar/zip files, let's wait a bit more for further feedback before adding it. :) In the worst scenario, we can make it an opt-in built-in step.
For whoever is reading this discussion, please note the bin/start
endpoint was dropped, because it was not extensible enough. So bin/my_app start
is the main way to boot it.
Here's the step I am using for making a tar file:
# ...
releases: [
my_app: [
include_executables_for: [:unix],
applications: [runtime_tools: :permanent],
steps: [:assemble, &make_tar/1]
]
],
# ...
defp make_tar(%Mix.Release{} = rel) do
tar_filename = "#{rel.name}.tar.gz"
out_path = Path.join(rel.path, tar_filename)
dirs =
["bin", "lib", Path.join("releases", rel.version), "erts-#{rel.erts_version}"] ++
[Path.join("releases", "COOKIE"), Path.join("releases", "start_erl.data")]
files = Enum.map(dirs, &{String.to_charlist(&1), String.to_charlist(Path.join(rel.path, &1))})
:ok = :erl_tar.create(String.to_charlist(out_path), files, [:dereference, :compressed])
:ok = File.rename(out_path, Path.join(rel.version_path, tar_filename))
rel
end
I used the same output path and tar options as https://github.com/bitwalker/distillery/blob/master/lib/distillery/releases/archiver/archive.ex because they seemed sane.
This is a meta issue to track the progress of releases in Elixir. The goal is to include minimum support for releases, then add user conveniences, and then finally work on hot code upgrades with relups and appups.
Milestone 1
The goal here is to add releases to core without many bells and whistles:
mix release
support to Elixir (#8677)Milestone 2
The goal here is implement user facing features:
Mix.Config
. The default config provider will look a config atconfig_path |> Path.dirname() |> Path.join("release.exs")
and always log if a release config is being used or not. We also need to raise a good error message ifMix.Config
is used instead ofConfig