Publish Nimbus packages

[zah]

Our build steps are already simple enough and unlikely to change significantly, so we can publish build recipes in package managers that build from source such as:

• [ ] Homebrew
• [ ] Gentoo
• [ ] NixOS
• [ ] FreeBSD

Other operating systems and package managers may be targeted with build scripts stored in this repo.

• [ ] FPM (can be used for creating packages for Debian, Ubuntu, CentOS, Fedora, etc)
• [ ] Chocolatey (for Windows users)

[arnetheduck]

not sure we want to go here too early - this will be a maintenance cost that isn't necessarily covered by user demand given that our main target group for now is developers that should work with source code

[zah]

I think we can treat this as a lower-priority bounty. My concern is that with such nice-to-have, but not mandatory features, the most likely outcome is that we'll be forced to cut them off as the release date becomes closer and we still have more important items to address.

If somebody steps in and builds the skeleton of these recipes, I think this can save us quite a bit of time, because researching these systems is 90% of the work. The required maintenance will be mostly about changing URLs and revision hashes and tweaking the build steps.

[bernardoaraujor]

how do I find out what are the artifacts that need to go into the package layout?

I built with make nimbus and I see the executable at build/nimbus. Is that all though? No config files or other artifacts that are needed at runtime?

[bernardoaraujor]

.deb + .rpm implemented via FPM on nimbus-eth2: https://github.com/status-im/nimbus-eth2/pull/2691

[jlokier]

In practice I think you do need ongoing maintenance of these build process and scripts, rather than "throw it over the wall" one-off development. The target OSes change often, different versions of the target OS also have quirks, testing is fairly important on each target, and what's built in Nimbus source also changes from time to time.

I agree that the research and knowledge is 90% of the effort, but then they break easily if not maintained. I would say the 90% should include creating a suitable reproducible build, deploy and test environment for each target.

For example we already have a Nix build script: https://github.com/status-im/nimbus-eth1/blob/master/nix/nim.nix

And yet, just from looking it looks like the version in the 2021 repo is only going to work on 2019 Nimbus. Multiply this by the number of target OSes and target OS versions and it looks like something that should be going through a regular Devops/CI process, rather than bit-rotting untested scripts.

[jlokier]

how do I find out what are the artifacts that need to go into the package layout?

I built with make nimbus and I see the executable at build/nimbus. Is that all though? No config files or other artifacts that are needed at runtime?

To directly answer your question, in the master branch right now I believe nimbus plus all of its shared library dependencies. The shared library dependencies vary depending on the build and OS, and some of them (librocksdb in the current version but it has varied) sometimes are not included in the ldd output.

Getting more philosophical, at the moment there isn't a correct forward-looking answer, because the source repo is not written with binary installation in mind. We never actually run nimbus except at its path build/nimbus. The library name in librocksdb.nim is potentially just wrong on some Linuxes; certainly it's not done the way it's supposed to be according to distro standards e.g. for .deb. And on MacOS ./build/nimbus doesn't run without additional linker environment variables to tell it where to find the shared libraries.

In one of my branches locally, build/nimbus requires build/libnimbus-evm.so and build/libnimbus-precompiles.so (on Linux; different names on other OSes), and they must be in the correct locations for ./build/nimbus to run, but whether you get that depends on a build option and of course which branch. We will almost certainly add other support files in time but not at the moment.

The other executables (for testing/support tooling) read files from hard-coded paths in the source tree, expecting those files to be present. I see we have some Dockerfiles (two) which install libraries that Nimbus isn't using any more (libpcre). This isn't surprising, but it highlights what happens when there are lots of different build scripts around, each containing their own copy of what the dependencies were at one time.

I would rather we don't end up with a collection of bit-rotting packaging scripts that contain a lot of hard-coded information. See the existing Nix files for lots of great examples of what to avoid: Hard-coded commit hashes, installs under a hard-coded version number (which needn't match elsewhere in the tree), refers to parts of the tree that don't exist any more.

I'd rather we end up with the Makefile or used-every-day build script which either outputs info for packaging scripts (such as list of shared libraries, or list of files and their type), or performs make install in a fairly standard way, using standard make variables such as prefix, execprefix etc. But of course that's a goal, we can get there in steps.

[bernardoaraujor]

Note: I should make it clear that the PR is active on nimbus-eth2. I referenced it here because @zah pointed me to this issue but then mentioned that this feature would make more sense for nimbus-eth2. Sorry if that caused any confusion.

In practice I think you do need ongoing maintenance of these build process and scripts, rather than "throw it over the wall" one-off development.

I'd be happy to provide ongoing maintenance on whatever approach this feature could end up being.

The target OSes change often, different versions of the target OS also have quirks, testing is fairly important on each target, and what's built in Nimbus source also changes from time to time.

Not sure I agree. The proposed package layout is quite simple and follows LSB FHS. And as long as the release executable has no dependencies on hard-coded paths in the source tree (which shouldn't happen IMO), there shouldn't be major issues.

For example we already have a Nix build script: https://github.com/status-im/nimbus-eth1/blob/master/nix/nim.nix And yet, just from looking it looks like the version in the 2021 repo is only going to work on 2019 Nimbus. Multiply this by the number of target OSes and target OS versions and it looks like something that should be going through a regular Devops/CI process, rather than bit-rotting untested scripts.

Indeed, I'm really aiming for something that will be processed by the CI workflow right before release, and provide packages ready to be installed under Debian, RHEL and SUSE derivatives. Perhaps even packages to be sent to upstream Distro Package Feeds or PPAs and COPRs.

I agree that the research and knowledge is 90% of the effort, but then they break easily if not maintained. I would say the 90% should include creating a suitable reproducible build, deploy and test environment for each target.

Maybe something like this written into the release workflow?

To directly answer your question, in the master branch right now I believe nimbus plus all of its shared library dependencies. The shared library dependencies vary depending on the build and OS, and some of them (librocksdb in the current version but it has varied) sometimes are not included in the ldd output.

Getting more philosophical, at the moment there isn't a correct forward-looking answer, because the source repo is not written with binary installation in mind. We never actually run nimbus except at its path build/nimbus. The library name in librocksdb.nim is potentially just wrong on some Linuxes; certainly it's not done the way it's supposed to be according to distro standards e.g. for .deb. And on MacOS ./build/nimbus doesn't run without additional linker environment variables to tell it where to find the shared libraries.

Indeed these are issues worth paying attention to. I believe it really comes down to how the CI server will be running the build and what runtime dependencies end up listed as a result. The solution can be quite straightforward though, since FPM does have a mechanism for declaring runtime dependencies, which prevent the package from being installed unless they're all met. However I must admit this is not 100% covered in my proposed solution yet, I'll make sure I work something out.

The other executables (for testing/support tooling) read files from hard-coded paths in the source tree, expecting those files to be present.

Although I did include some tooling executables on my initial solution, I'm removing them as a suggestion from @stefantalpalaru. Only nimbus_beacon_node and nimbus_signing_process will be packaged.

I would rather we don't end up with a collection of bit-rotting packaging scripts that contain a lot of hard-coded information. See the existing Nix files for lots of great examples of what to avoid: Hard-coded commit hashes, installs under a hard-coded version number (which needn't match elsewhere in the tree), refers to parts of the tree that don't exist any more.

I'm making sure that does not happen. I'm not introducing any extra steps into the build process. The packaging script assumes building is already finished.

I'd rather we end up with the Makefile or used-every-day build script which either outputs info for packaging scripts (such as list of shared libraries, or list of files and their type), or performs make install in a fairly standard way, using standard make variables such as prefix, execprefix etc. But of course that's a goal, we can get there in steps.

My end goal is to provide pre-built and pre-packaged binaries. Something the end-user can just do dpkg -i nimbus_xxx.deb or dnf install nimbus_xxx.rpm. And perhaps even apt-get install nimbus + apt-get upgrade nimbus in case they end up in some Package Feed.

make install is a different approach, it assumes the end-user is building from source. A viable path, but kills the Package Feed possibility and sets a relatively higher bar for the end-user (git clone, build Nim).

I'm proposing that the used-every-day script would be the release workflow. In case create-packages.sh is polluting the repository, it's underlying logic could alternatively just be embedded into the release script instead. I'm not really married to the standalone create-packages.sh script, although I feel it's useful for manual generation of packages and not limited to release scenarios only.