Explore if we can replace ockam-builder docker image with Nix

[mrinalwadhwa]

Current

We use a docker image ockam-builder to get a near reproducible build environment https://github.com/build-trust/ockam/blob/develop/tools/docker/builder/Dockerfile

The is used in many of our Github Workflows https://github.com/build-trust/ockam/tree/develop/.github/workflows

And is available as a devcontainer that can launch various IDEs and code spaces https://github.com/build-trust/ockam/blob/develop/.devcontainer/Dockerfile

Desired

Could we use Nix for this instead? What would be the tradeoffs?

[etorreborre]

One approach could be to use flox and nix to install all the tools, then everything is configured and available on the path. I updated the ockam flox environment here: https://github.com/etorreborre/floxmeta/tree/aarch64-darwin.ockam. That environment contains:

❯ flox list -e ockam                                                                                                                                                                                                                                                                                                                                                                                        ockam-dev go .#telegraf default
etorreborre/aarch64-darwin.ockam
    Alias     ockam
    System    aarch64-darwin
    Path      /Users/etorreborre/.local/share/flox/environments/etorreborre/aarch64-darwin.ockam
    Curr Gen  61

Packages
    0   stable.nixpkgs-flox.apacheKafka@2.13-3.3.1  apache-kafka-2.13-3.3.1
    1   stable.nixpkgs-flox.bats                    1.8.2
    2   stable.nixpkgs-flox.cmake                   3.25.1
    3   stable.nixpkgs-flox.commitlint              _at_commitlint_slash_cli-17.4.2
    4   stable.nixpkgs-flox.cosign                  1.13.1
    5   stable.nixpkgs-flox.eclint                  0.3.8
    6   stable.nixpkgs-flox.elixir                  1.14.3
    7   stable.nixpkgs-flox.erlang                  25.2.3
    8   stable.nixpkgs-flox.gradle@7.4.2            7.4.2
    9   stable.nixpkgs-flox.influxdb2               influxdb-2.0.6
    10  stable.nixpkgs-flox.jdk@17.0.3              zulu17.34.19-ca-jdk-17.0.3
    11  stable.nixpkgs-flox.jq                      1.6-bin
    12  stable.nixpkgs-flox.mold                    1.10.1
    13  stable.nixpkgs-flox.nodejs                  18.14.1
    14  stable.nixpkgs-flox.pipenv                  2023.2.4
    15  stable.nixpkgs-flox.postgresql_15           postgresql-15.2
    16  stable.nixpkgs-flox.python3Full             python3-3.10.9
    17  stable.nixpkgs-flox.rustup                  1.25.2
    18  stable.nixpkgs-flox.shfmt                   3.6.0
    19  stable.nixpkgs-flox.terragrunt              0.43.2
    20  staging.nixpkgs-flox.telegraf@1.26.0        1.26.0

Building and loading a container out of this list is as simple a running

flox containerize -e ockam | docker load

Now the devil is in the details and we would have to check if all the dependencies are correct and if everything is available. For instance I think that the bats-support and bats-assert libraries are missing. But then it shouldn't be too hard to publish them as nix packages.

[etorreborre]

Some possible trade-offs in my mind.

Pros:

• nix is a robust build system and nixpkgs is a very large collection of packages
• it not only contains libraries and executables but full toolchains
• it is not hard to publish new packages in our own collection of packages if necessary
• the same environment can be installed (and instantly removed) on any developer machine for good reproducibility without having to enter a container

Cons:

• nix is a new language to learn. Resources are getting better but still it is a peculiar configuration language
• issues can be hard to debug if you are new to nix
• I personally have only been using nix with flox because of the very nice UI and the possibility to segregate and share environments. If we were to use flox as well, that would be yet another tool to manage (I would still recommend it though)
• I haven't tested the containerized images nor a flox environment to support or workflows so there might be unforeseen and difficult issues to solve

[etorreborre]

An alternative to flox that I haven't tried (I am not even sure if they totally overlap) is https://devenv.sh

[mrinalwadhwa]

In https://github.com/build-trust/ockam/pull/4902 @shanesveller added nix based environment.

[etorreborre]

This has been done via the work of @metaclips: https://github.com/build-trust/ockam/pull/5748 and #5843