search

NixOS / nixops

NixOps is a tool for deploying to NixOS machines in a network or cloud.
https://nixos.org/nixops
GNU Lesser General Public License v3.0
1.82k stars 363 forks source link

Deploying from nix-shell under root #319

Open danbst opened 9 years ago

danbst commented 9 years ago

I was doing this:

  1. install nixos on ec2 from nixos AMI
  2. git clone nixops on ec2 machine
  3. run ./dev-shell
  4. nixops deploy
building path(s) в??/nix/store/6slz3xj7ski2jyza20m27l6dg85wyysy-system-unitsв?T
/nix/store/jm26xg0h3jcrg4bbrwiqx3jpirscdk0p-stdenv/setup: line 456: /run/user/0/nix-build-system-units.drv-0/env-vars: Permission denied
builder for в??/nix/store/sjhxwnljr0bh8rwjvqawcwypir82rqci-system-units.drvв?T failed with exit code 1
cannot build derivation в??/nix/store/gv7fsdkh6x1rmqfh1v7jhr4m8p6k7l8z-etc.drvв?T: 1 dependencies couldn't be built
cannot build derivation в??/nix/store/j1k7ggpaswv2v4kx5if8fq46kzm7rdc5-nixos-15.06pre64647.a73d25d.drvв?T: 1 dependencies couldn't be built
cannot build derivation в??/nix/store/gjk9p7hbrv95kjf9pf878sfd9j7xy9mf-nixops-machines.drvв?T: 1 dependencies couldn't be built
error: build of в??/nix/store/gjk9p7hbrv95kjf9pf878sfd9j7xy9mf-nixops-machines.drvв?T failed
error: unable to build all machine configurations

in IRC we concluded that the problem is in temp directory and solution is to

mkdir /tmp/foo
export TMPDIR=/tmp/foo

This is quite non-obvious behaviour for those, who want to hack on nixops, and when running nixops deploy without dev-shell, everything is ok.

nh2 commented 7 years ago

I'm suffering from this, because I use nixops from a nixops clone directly. Workaround works though.

@aszlig is this tag backends/ec2 right?

aszlig commented 7 years ago

@nh2: Hm, doesn't seem so, I'll remove the label.

aszlig commented 7 years ago

So am I correctly assuming that $TMPDIR is /run/user/0 here?

nh2 commented 7 years ago

It seems the TMPDIR workaround is not enough:

While it fixes the error message appearing, it corrupts files in the nix store. Some just end up empty (0 bytes) when really they should not be.

For example, when deploying I get the error:

/nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml:1: parser error : Document is empty

And indeed that file is empty. But the same path is not empty on my Ubuntu machine (where I also run nix-shell -p nixops, but not as root) and has sensibel contents there.

@cleverca22 told me how I can spot the reason:

On the good (Ubuntu) machine:

nix-store -l /nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml has no output.

But on the problematic machine:

# nix-store -l /nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml
/nix/store/685n25b9yc8sds57vljk459ldly1xyhn-stdenv/setup: line 500: /run/user/0/nix-build-grub-config.xml.drv-0/env-vars: Permission denied

So that's our error again!

@aszlig This also mentions the /run/user/0 that you were mentioning.

danbst commented 7 years ago

Also, I don't know how to fix your error (I don't run nixops from under nix-shell from under root anymore, I actually rebuild it after changes), I have encountered those errors outside of nixops context:

  1. I had a zero-length file in store, which happend to be used in /boot
  2. I often had /tmp/env-vars: Permission denied when doing nix-shell builds with two different users. In particular, sometimes /tmp/env-vars is created by user deployer, and user danbst can't do any builds in nix-shell anymore (insufficient rights to overwrite /tmp/env-vars).

So now I think, this isn't nixops issue, but a Nix one.

dhruvio commented 6 years ago

For posterity, see https://github.com/NixOS/nix/issues/568#issuecomment-348312937