search

cachix / devenv

Fast, Declarative, Reproducible, and Composable Developer Environments
https://devenv.sh
Apache License 2.0
4.45k stars 334 forks source link

Allow shipping containers from macOS #430

Open domenkozar opened 1 year ago

domenkozar commented 1 year ago

Please describe your use case how you'd like to use containers from macOS.

jmuchovej commented 1 year ago

I'm new to devenv (and nix for that matter), but this seems like an enticing way to manage dependencies and the like! 🙂

I'm usually on MacOS, so I'm happy to test (and potentially dev over the next few weeks), though given my ignorance around nix, I'll probably have myriad questions. 😅

Anywho – I just tried using containers on MacOS (to see what happens), and I received the following as output:

$ devenv container shell
~/.nix-profile/bin/devenv: line 110: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Based on this SO answer, it seems that declare -A requires Bash 4+, but MacOS ships with Bash 3. However, MacOS now uses zsh by default. From my limited understanding, zsh has associative arrays – though I've read that they're "... implemented differently ..." than associative arrays in bash (no idea how).

Looking at the "offending line", the call to /nix/store/m76vd4ma4455s1rsqm8kq8ps5vhinvmj-docopts-0.6.4-with-no-mangle-double-dash/bin/docopts -A subcommand -h "$help" : "$@" is where the problem arises since docopts targets Bash 4+, though docopts enumerates how to achieve Bash <4 compatibility. To me, this points to a few routes forward for MacOS support:

  1. Implement some dynamic checking to permit associative arrays (some sort of switch on MacOS that removes -A and implements the other compatibility changes)
  2. Install bash 4+ via nix? (I have no idea if this is possible.)
  3. Use something other than docopts.

Edit: Removing info about ZSH, since docopts doesn't support it, so it's probably a non-starter. 😅

jmuchovej commented 1 year ago

Update: I installed bash via nix-env --install bash (which showed up as 5.2-p15 for me). After installing bash, I was able to get to container "compilation" (not sure what to call it), once I added the nix2container bits to my devenv.yaml.

At the moment, I'm now "stuck" here:

$ devenv container devcontainers --docker-run
/nix/store/0ggk9l9spp3gfcl995idcwf3pbb3zhyg-image-devcontainers.json

Copying container /nix/store/0ggk9l9spp3gfcl995idcwf3pbb3zhyg-image-devcontainers.json to docker-daemon:devcontainers:latest

Getting image source signatures
Copying blob 606ff57294bb done
FATA[0004] writing blob: io: read/write on closed pipe

Even when leaving containers."devcontainers"... unspecified (as I did prior to this), I get the same errors. I'm not too certain how to proceed with debugging from here, so some pointers would definitely be helpful! 🙂

For reference, this is my devenv.nix and devenv.yaml:

devenv.nix ```nix { pkgs, config, ... }: { name = "devcontainers"; devcontainer.enable = true; processes = { ping.exec = "ping 8.8.8.8"; }; containers."devcontainers".name = "devcontainers"; containers."devcontainers".startupCommand = config.processes.ping.exec; # https://github.com/cachix/devenv/blob/main/examples/starship/devenv.nix starship.enable = true; starship.config.enable = true; languages.javascript.enable = true; languages.typescript.enable = true; languages.go.enable = true; # https://devenv.sh/packages/ packages = [ pkgs.zsh pkgs.git ]; } ```
devenv.yaml ```nix inputs: nixpkgs: url: github:NixOS/nixpkgs/nixpkgs-unstable nix2container: url: github:nlewo/nix2container inputs: nixpkgs: follows: nixpkgs mk-shell-bin: url: github:rrbutani/nix-mk-shell-bin ```
domenkozar commented 1 year ago

The main issue is that the container needs to be built on a Linux machine.

aflatter commented 1 year ago

Please describe your use case how you'd like to use containers from macOS.

Hum, I keep having trouble with direnv + VSCode, so I'd like to try devcontainer with devenv. Unfortunately I'm on a macOS machine so I can't build the container.

willmruzek commented 1 year ago

I would like to be able to build images on MacOS for aarch64-linux and x86_64-linux for:

  1. Build images for local dev (cross-platform env Mac|WSL2|Linux)
  2. Build images for remote dev (SSH, Github Codespaces, GitPod)
  3. Build images for staging and production

(MacOS and WSL2 are my main dev envs.)

Can we do all this within a Linux container on Mac for Docker already? Just curious.

Looking forward to something like this!

domenkozar commented 1 year ago

Yeah, you can do it in a container or WSL2.

We could support cross-compiling, but that would limit to only packages that support cross-compilation (which is still a lot).

willmruzek commented 1 year ago

Good to know, thanks.

Cross-compiling would be important for my case since I'm on Apple Silicon, but most cloud/server CPUs are x86_64.

mrfelton commented 1 year ago

Yeah, you can do it in a container

Are there any instructions for how to do this?

I'd like to build a devcontainer for use with VS Code on a Mac.

devenv container shell --docker-run does not work for me. Though I get a different error to what @jmuchovej posted. I get:

❯ devenv container shell --docker-run
/nix/store/6mil5af39zlgxaplzpqnjfl80r89qzfj-image-zap-devenv-shell.json

Copying container /nix/store/6mil5af39zlgxaplzpqnjfl80r89qzfj-image-zap-devenv-shell.json to docker-daemon:zap-devenv-shell:latest

Getting image source signatures
Copying blob 85f15165725a done
Copying config 3910136d57 done
Writing manifest to image destination
Storing signatures
exec /nix/store/cmybmvzb77znaay66f0lqb2s20rd7zim-entrypoint: exec format error

If I try to open the devenv in VC Code and start a VS Code terminal with direnv, I get:

direnv: loading /workspaces/workspace/.envrc
direnv: loading https://raw.githubusercontent.com/cachix/devenv/d1f7b48e35e6dee421cfd0f51481d17f77586997/direnvrc (sha256-YBzqskFZxmNb3kYVoKD9ZixoPXJh1C9ZvTLGFRkauZ0=)
direnv: using devenv
direnv: .envrc changed, reloading
Building shell ...
evaluating derivation 'git+file:///workspaces/workspace#devShells.x86_64-linux.default'direnv: ([/nix/store/xwsp5wk6141wf5zskld99s1c36wmh4z7-direnv-2.32.2/bin/direnv export bash]) is taking a while to execute. Use CTRL-C to give up.
error:
       … while setting up the build environment

       error: unable to load seccomp BPF program: Invalid argument
./.envrc:1389: pop_var_context: head of shell_variables not a function context
environment:1: pop_var_context: head of shell_variables not a function context

If I enable devcontainers (devcontainer.enable = true;) and try to open the devenv in VC Code by opening the workspace with devcontainer.json, I get a similar error on building when VSCode builds the devcontainer:

error:
       … while setting up the build environment

       error: unable to load seccomp BPF program: Invalid argument
[338769 ms] updateContentCommand failed with exit code 1. Skipping any further user-provided commands.
[338773 ms] Error: Command failed: /bin/sh -c devenv ci
[338773 ms]     at IN (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1733:130)
[338773 ms]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[338773 ms]     at async E1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:4370)
[338773 ms]     at async p1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:3727)
[338773 ms]     at async g1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:2815)
[338774 ms]     at async Bae (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:2092:29064)
[338774 ms]     at async Sae (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:2092:26423)
[338775 ms] Exit code 1
[338776 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) --ms-enable-electron-run-as-node /Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js run-user-commands --user-data-folder /Users/tom/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --container-session-data-folder /tmp/devcontainers-57901f0a-990a-49a9-a5b1-3e15b1f3f1311683630384316 --workspace-folder /Users/tom/workspace --id-label devcontainer.local_folder=/Users/tom/workspace --id-label devcontainer.config_file=/Users/tom/workspace/.devcontainer.json --container-id cb7d09269e077bd750d53888e8c9762766ad36f8532e38d18bb0a43103bf1f76 --log-level debug --log-format json --config /Users/tom/workspace/.devcontainer.json --default-user-env-probe loginInteractiveShell --skip-non-blocking-commands true --prebuild false --stop-for-personalization true --remote-env REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-29878c87-f6bd-4f8e-bfba-1651c1fb1c44.sock --remote-env SSH_AUTH_SOCK=/tmp/vscode-ssh-auth-29878c87-f6bd-4f8e-bfba-1651c1fb1c44.sock --remote-env REMOTE_CONTAINERS=true --mount-workspace-git-root true --terminal-columns 182 --terminal-rows 29 --dotfiles-target-path ~/dotfiles
nathanscully commented 1 year ago

Please describe your use case how you'd like to use containers from macOS.

Would like to see if it could replace a local + docker-compose for a more performant native dev experience. Example use case:

  1. Run a rails/django/node/{etc} service locally that has service dependencies like Redis and Postgres.
  2. Be able to build and execute tests in a container like-for-like with a prod image that would be used for deployment. The concern I see with the local only + local services is if you push code your remote ci/cd and deployed image could have issues you don't see running locally.
  3. Run and execute the local service directly in a container (with the accepted performance hit) to replicate the same architecture as a deployed image. This saves pulling a built image down from your ci/cd stack to debug issues.

My alternative at the moment is running process dependencies in docker containers and managing them via docker-compose, or even the local service at a performance impact due to the docker or podman VM layer. But it gives you more confidence avoiding the works on my machine issue.

tl;dr local dev with service/process dependencies that we can test in the same "prod" deployment environment without the macOS container VM overhead.

If this is possible somehow already would appreciate someone sharing a walkthrough!

domenkozar commented 1 year ago

The easiest way to achieve this is using linux builder on macos: https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

domenkozar commented 1 year ago

Please describe your use case how you'd like to use containers from macOS.

Would like to see if it could replace a local + docker-compose for a more performant native dev experience. Example use case:

  1. Run a rails/django/node/{etc} service locally that has service dependencies like Redis and Postgres.
  2. Be able to build and execute tests in a container like-for-like with a prod image that would be used for deployment. The concern I see with the local only + local services is if you push code your remote ci/cd and deployed image could have issues you don't see running locally.
  3. Run and execute the local service directly in a container (with the accepted performance hit) to replicate the same architecture as a deployed image. This saves pulling a built image down from your ci/cd stack to debug issues.

My alternative at the moment is running process dependencies in docker containers and managing them via docker-compose, or even the local service at a performance impact due to the docker or podman VM layer. But it gives you more confidence avoiding the works on my machine issue.

tl;dr local dev with service/process dependencies that we can test in the same "prod" deployment environment without the macOS container VM overhead.

If this is possible somehow already would appreciate someone sharing a walkthrough!

That's currently possible using https://devenv.sh/services/ and https://devenv.sh/containers.

domenkozar commented 7 months ago

I'd be great if someone can sketch out how to setup a linux builder and then run devenv --system aarch64-linux container build project.

kvz commented 7 months ago

My use case: I'd like to test if containers work at all locally on my Apple Sillicon, before handing things over to CI and prod. Potentially, I'm shipping these containers to production also already as I'm still early stages. It's fine for me if the resulting containers are ARM linux ones, I'll just run them on graviton (or by the time I involve CI, it will produce x64 counterparts. The docs say that:

To be able to generate containers on macOS, you will need to use a remote Linux builder. The easiest ways to do so are: - Set the remote builder up using Nix. - Use the nix-darwin linux-builder module.

So I followed the former link's instructions and have a linux builder running, and I adapted my /etc/nix/nix.conf and /etc/ssh/ssh_config.d/100-linux-builder.conf accordlingly

<<< Welcome to NixOS 24.05pre-git (aarch64) - ttyAMA0 >>>

Now when I run devenv container build processes I would expect the linux builder to produce an arm linux container with all the processes inside, instead it errors out:

λ devenv container build processes
Error:   × Containers are not supported on macOS yet: https://github.com/cachix/devenv/issues/430

λ devenv --system aarch64-linux container build processes
Error:   × Containers are not supported on macOS yet: https://github.com/cachix/devenv/issues/430

How would I get past this check so the linux-builder is used?

I'm on 1.0.3 by the way.

awooff commented 1 month ago

Bumping this, is there any update on this issue? I too would also like to be able to use this feature - I use Nix in all my (and work's) infra and not having to use WSL thru a VM just to be able to work on this would be indispensible.

marcusramberg commented 2 weeks ago

@domenkozar I have a linux builder on my mac, but even with devenv --system aarch64-linux container build project I get Error: × Containers are not supported on macOS yet: https://github.com/cachix/devenv/issues/430