search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.39k stars 275 forks source link

[FEATURE]: Add package for Nix/NixOS #4935

Open sagikazarmark opened 1 year ago

sagikazarmark commented 1 year ago

Feature Request

Background / Motivation

I use Nix as my package manager for macOS and Linux.

What should the user be able to do?

Install garden using Nix.

Why do they want to do this? What problem does it solve?

Nix is particularly useful in creating portable development environments. Since garden is often used for development, integrating it into such environments is crucial.

Suggested Implementation(s)

Create a Nix package.

How important is this feature for you/your team?

šŸŒµ Not having this feature makes using Garden painful

Additional information

There is already a closed issue asking for a Nix package: #2785

I decided to open a new issue, because I'd be happy to do the work, but I'd need some assistance because I'm not intimately familiar how Node CLIs are packaged (particularly garden seems to require some additional files for it to run).

What I need to be able to create a Nix package:

I tried following the contributing guide, but it's not obvious what the output of yarn build is.

Any assistance would be greatly appreciated.

Thanks!

sagikazarmark commented 1 year ago

Here is an initial version using the binary distribution for now: https://github.com/sagikazarmark/nix-garden

stefreak commented 1 year ago

@sagikazarmark Cool, thank you for working on this!

In our contributing guide we have a [section about how to build the release binaries and docker containers]().

In NixOS you use glibc by default, right?

Basically we need the built pkg vercel nodejs binary, and a "static" directory (that needs to be initialized as a git repository unfortunately right now).

See also the debian Dockerfile: https://github.com/garden-io/garden/blob/main/support/buster.Dockerfile#L47C2-L47C2 The build context root for that Dockerfile dist/linux-amd64 (This directory exists after running yarn dist)

Happy to help if you run into problems.

stefreak commented 1 year ago

Unfortunately we have no official ARM binaries yet, which should be straightforward to change nowadays but we are tracking a separate issue for that: #1547

sagikazarmark commented 1 year ago

@stefreak thanks for getting back to me.

As far as I understand the value behind building the vercel pkg binaries is to distribute garden as a single executable. I'm not sure the Nix package needs that as Nix is perfectly capable of installing Node as a dependency as well.

The repo I linked above already uses the binary distribution which is a good start IMO, but I'd like to play a bit with skipping the final vercel build and just use ... the artifact that comes before that. A script file? Not sure yet.

In NixOS you use glibc by default, right?

Yep

Basically we need the built pkg vercel nodejs binary, and a "static" directory (that needs to be initialized as a git repository unfortunately right now).

One problem I already have (with the binary distributed version) is this:

Unexpected Git error occurred while running 'git status' from path "/nix/store/hlg35xcp6aihwazrcvy5s8b9b63hm0bh-garden-0.13.10/static". Exit code: 128. Error message: fatal: detected dubious ownership in repository at '/nix/store/hlg35xcp6aih
wazrcvy5s8b9b63hm0bh-garden-0.13.10/static'
To add an exception for this directory, call:

        git config --global --add safe.directory /nix/store/hlg35xcp6aihwazrcvy5s8b9b63hm0bh-garden-0.13.10/static

Command "git status" failed with code 128:

   fatal: detected dubious ownership in repository at '/nix/store/hlg35xcp6aihwazrcvy5s8b9b63hm0bh-garden-0.13.10/static'
To add an exception for this directory, call:

        git config --global --add safe.directory /nix/store/hlg35xcp6aihwazrcvy5s8b9b63hm0bh-garden-0.13.10/static

Nix puts all packages in a so-called "Nix store" (basically /nix in the filesystem) and makes them read-only and owned by root, to make sure no accidental modifications happen.

I'll take a look at the links you've provided and come back if I have any questions. Thanks!

stefreak commented 1 year ago

@sagikazarmark then it might be possible that https://github.com/garden-io/garden/pull/4047 has to be finished first, where we get rid of the requirement that the static directory is a git repository.

sagikazarmark commented 1 year ago

@stefreak I managed to work around the problem for now: https://github.com/sagikazarmark/nix-garden/blob/main/flake.nix#L93-L98

The binary distribution seems to be working for the moment, so I'll test it in the next couple weeks and then submit it to the official Nix repository.

I'll keep working on the vercel-less version in the meantime.

worldofgeese commented 1 year ago

@sagikazarmark as a user of Nix and Guix this is something I've wanted for a long time (I created an issue in nixpkgs but never got around to filling).

Since I use Guix System on my personal laptop, I created a Guix package for garden over the weekend. ldd shows missing libraries on the raw garden binary:

ldd garden 
    linux-vdso.so.1 (0x00007ffcbbfe9000)
    libdl.so.2 => /gnu/store/ip9mj1pwymxi1yq32zbhwp3n3bycy6yi-glibc-2.35/lib/libdl.so.2 (0x00007fd2f5496000)
    libstdc++.so.6 => not found
    libm.so.6 => /gnu/store/ip9mj1pwymxi1yq32zbhwp3n3bycy6yi-glibc-2.35/lib/libm.so.6 (0x00007fd2f53b9000)
    libgcc_s.so.1 => not found
    libpthread.so.0 => /gnu/store/ip9mj1pwymxi1yq32zbhwp3n3bycy6yi-glibc-2.35/lib/libpthread.so.0 (0x00007fd2f53b4000)
    libc.so.6 => /gnu/store/ip9mj1pwymxi1yq32zbhwp3n3bycy6yi-glibc-2.35/lib/libc.so.6 (0x00007fd2f51b6000)
    /lib64/ld-linux-x86-64.so.2 => /gnu/store/ip9mj1pwymxi1yq32zbhwp3n3bycy6yi-glibc-2.35/lib/ld-linux-x86-64.so.2 (0x00007fd2f549d000)

My Guix package definition is unfinished because patchelf-ing in these library paths changes the binary size, which Pkg has an issue with because it uses hardcoded offsets. As far as I can tell the solution is to manually bodge the PAYLOAD_POSITION and PRELUDE_POSITION as this packager did in 2018.

I've noticed you don't use patchelf. I'm confused why your flake works for you because as I understand NixOS suffers from the same issue as Guix System in needing to patch the RPATH and dynamic linker of most binaries to work with these distros. How have you avoided it?

In any case, thanks for your efforts to package garden for Nix! This will open up our tool to be used across a broader range of non-FHS compliant distros and by macOS users that don't want to deal with Homebrew (far and away the worst package manager I've ever used).

sagikazarmark commented 1 year ago

@worldofgeese I only used the package on macOS so far, so that might answer why it works.

The garden binary itself is packaged using vercel/pkg which I think we should be able to omit (that could also make packaging easier).

worldofgeese commented 1 year ago

@sagikazarmark I spoke with @TimBeyer to understand our binary build process a little more: yarn build and yarn dist call out to https://github.com/garden-io/garden/blob/main/cli/src/build-pkg.ts. Tim tells me it'd be non-trivial to duplicate what build-pkg.ts does for another packager. However, we did arrive at some promising possibilities!

  1. yarn install is not an alias but a standard yarn command and installs all dependencies into the local node_modules folder. If we pull in yarn as an explicit dependency we can use this in our package definitions to make an FHS-independent package that runs on NixOS and Guix System.
  2. We used to publish garden to NPM but stopped. If we began publishing to NPM again, we could just install garden as an NPM package within our build environments.

There are performance penalties to using Pkg in the resulting binary so it may be we see performance improvements by pursuing either of these alternatives in our own definitions.

sagikazarmark commented 1 year ago

@worldofgeese these are great news! I could see either of these happening.

  1. maybe slightly easier from a Nix/Guix perspective, but puts more requirements on garden. 1. is a bit more work on the Nix side, but makes less assumptions about packaging.

I believe we should pursue both (given publishing on NPM is something you can/want to do), but for the long term, option 1 is probably a better solution.

There are performance penalties to using Pkg in the resulting binary so it may be we see performance improvements by pursuing either of these alternatives in our own definitions.

I definitely see that penalty on macOS (especially due to the Rosetta requirement).

TimBeyer commented 1 year ago

I definitely see that penalty on macOS (especially due to the Rosetta requirement).

We just published a release with native ARM binaries for macOS which has much improved performance over the version running via Rosetta. There's still a small overhead for running in pkg but overall it's a much better experience. I recommend you try it out!

sagikazarmark commented 1 year ago

@TimBeyer thanks! I upgraded to the new version: https://github.com/sagikazarmark/nix-garden/commit/8d05717d369ed55a06bc294e51142fd477d23508

It is indeed somewhat better.

Looking at the latest release note I noticed there is a self-update option. @worldofgeese I think it would be nice if we could disable it somehow in the nix packaged version.

worldofgeese commented 1 year ago

@sagikazarmark I'm working on this after work hours so I won't be the fastest person in the world but I made some good progress today! I found a yarn package definition for Guix (make sure it's 1.* because we're not compatible with 2), built it, opened a shell into its definition (guix shell -f yarn.scm), cloned the garden repo, ran yarn build && yarn install (you need to build to get some of the dependencies like open-tracing) then ran cli/bin/garden and... it works! Or at least the CLI shows me a list of arguments: so far so good.

Next steps will be to compose these steps into a definition, which I've already started on.

sagikazarmark commented 1 year ago

@worldofgeese I also started to work on a definition here: https://github.com/sagikazarmark/nix-garden/commit/b621f9ff5c7f01266da23d865d4a33f42561cf51

Unfortunately, yarn build fails with the following error:

[1/0/3 built, 0.0 MiB DL] building garden (configurePhase): configuringdirenv: ([/nix/store/2fq15zs5fq3ag5ynfxxaa3nw6ys0z6i9-direnv-2.32.3/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
error: builder for '/nix/store/fs4p6223viqjdcgkh1cillma9jw61k2y-garden.drv' failed with exit code 1;
       last 10 log lines:
       > core ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„  $ ./scripts/run-script.ts clean && find . -name ".garden" -type d -prune -exec rm -rf '{}' '+'
       > core ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„  SyntaxError: Unexpected token  in JSON at position 0
       > core ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„  error Command failed with exit code 1.
       > core ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„ā”„  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
       > 
       > clean script in package @garden-io/core failed with code 1
       > error Command failed with exit code 1.
       > info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
       > error Command failed with exit code 1.
       > info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
       For full logs, run 'nix log /nix/store/fs4p6223viqjdcgkh1cillma9jw61k2y-garden.drv'.

Unfortunately, it doesn't tell more about what the problem is.

It could be some automagic, that mkYarnPackage does (I noticed for example that garden utilizes workspaces), but I haven't been able to figure out the problem so far.

I have two suggestions for making progress at this point:

  1. Submit a PR with my binary based definition so that we can get garden in nixpkgs. I know such PRs have been accepted in the past.
  2. Let's join forces and work on a common repo (I'm happy to add you to mine, but I'm okay with working on whatever repo (eg. under garden) as well).

I understand the time constraints, so I think it makes sense to not duplicate efforts.

What do you think?

worldofgeese commented 1 year ago

@sagikazarmark I built a garden Guix package definition that uses yarn to build but ran into the wrinkle that Guix build environments are not network-capable.

Looking at your Nix package and based on mkYarnPackage's Nix definition it looks like it takes a version argument. Could you try and pass any version of yarn version 1 to it? garden will only build with v1.

In any case, based on our shared difficulties, I suggest creating an NPM package of garden, then using Nix and Guix's native Node build methods to source it. I was going to try to yarn publish to my own NPM namespace now and see if that works and if not, investigate what we'd need to get a functional NPM package. To that end, @stefreak indicated the static directory and the nodejs options that we need to control may be tricky to solve for. Once they are, it should be trivial to package for Nix and Guix.

Happy to work on any common repo with the objective of getting these accepted into their respective homes (Nixpkgs and nonguix)!

sagikazarmark commented 1 year ago

@worldofgeese great news!

Since the binary distribution works rather well at the moment, I think it makes sense to wait for an NPM package instead of struggling to come up with a build definition (that may never work).

One thing that doesn't work with the NPM definition is disabling self-update though (which is a bit unfortunate, but not the end of the world).

stefreak commented 1 year ago

@sagikazarmark your original approach should work fine, can you maybe dig deeper into why clean fails with that JSON error?

run-script.ts clean basically runs yarn clean in every single package directory. To do that, it reads the definition of the clean task from the package.json.

The output contains the log line clean script in package @garden-io/core failed with code 1 which suggests that it tried to read the package.json In core/.

Does the core/package.json contain a weird character at the beginning of the file? Can you share with us the output of hexdump -C core/package.json?

worldofgeese commented 1 year ago

@stefreak @sagikazarmark I have a PR I'm hoping to get up end of day today that adds an NPM package to simplify this process. You can already try it with npx @worldofgeese/cli. I will write back here when it's up for review.

I've tested with my own projects and it works great for all commands I've tested. In fact, it is significantly faster in all operations.

EDIT: it's up without any context. I will add more details later tonight or by the morning.

stefreak commented 1 year ago

We re-did how we build Garden and implemented a new binary shipping mechanism (garden-sea)

We now implement some environment variables in this wrapper binary: https://github.com/garden-io/garden/blob/main/garden-sea/src/node.rs#L43C12-L43C12 Namely GARDEN_MAX_SEMI_SPACE_SIZE, GARDEN_MAX_OLD_SPACE_SIZE, and GARDEN_SEA_UV_USE_IO_URING.

We also updated Node.js to 21.1 in the mean time. Should we try again with the npm run dist approach (https://docs.garden.io/v/edge-release/contributing-to-garden/developing-garden#release-binaries-and-docker-containers) or do we need to go down the NPM package path? With the NPM package we'd have less control over NodeJS versions and these kind of low level NodeJS settings.

stefreak commented 1 year ago

If we want to go down the NPM package path, #4477 is resolved now so the NPM PR is unblocked and we can continue working on it.