search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.19k stars 13.47k forks source link

Support user-data in minimal ISO image (Was: Add cloud-init service to minimal ISO) #39076

Open wmertens opened 6 years ago

wmertens commented 6 years ago

There are other cloud providers than AWS that support the userdata system.

Since the minimal ISO works everywhere, I propose that the ISO gets services.cloud-init.enabled = true;. If there is no config source, nothing will happen, and thus the minimal ISO can be used to bootstrap on cloud services.

wmertens commented 6 years ago

Looking at http://cloudinit.readthedocs.io/en/latest/topics/examples.html and the fact that the AWS ones can just be bash scripts, there is no real standard. It seems that you can provide any data you like.

So for a NixOS deployment-on-a-cloud-using-ISO scenario, what would be good information to expect in user-data? I'm thinking:

It would be kind of cool to have user-data just be a Nix file and it provides attributes for all the above, which would be evaluated in a way that would allow partitioning the disk and installing to /mnt.

Maybe a derivation that builds an install script that is run when it built successfully?

/cc @Mic92 @copumpkin

nlewo commented 6 years ago

I'm currently using cloud-init to push ssh-keys but I think it would be better to push a configuration.nix and rebuild the system. There is a cloud-init module to write a file and another one to run a command. So, I think via user-data, we could write a configuration.nix file a run nixos-rebuild switch. But I never tryied.

Do you know the size of the cloud-init closure? Which cloud provider or IAAS stack are you using?

wmertens commented 6 years ago

On Wed, Apr 18, 2018 at 11:10 AM lewo notifications@github.com wrote:

I'm currently using cloud-init to push ssh-keys but I think it would be better to push a configuration.nix and rebuild the system. There is a cloud-init module to write a file and another one to run a command. So, I think via user-data, we could write a configuration.nix file a run nixos-rebuild switch. But I never tryied.

Do you know the size of the cloud-init closure? Which cloud provider or IAAS stack are you using?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/39076#issuecomment-382319488, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWluIeekwClERvtY8L5YQMUYdYz7k9ks5tpwMLgaJpZM4TYkgY .

coretemp commented 6 years ago

@wmertens Good idea. When work on this is done, clarifying the documentation by mentioning http://cloudinit.readthedocs.io/en/latest/ would be helpful too then. Perhaps including a list of cloud providers in the documentation which support cloudinit would also be useful. Perhaps the cloudinit project makes such a list available.

edolstra commented 6 years ago

Adding 60 MB to the minimal ISO is not really compatible with it being minimal...

edolstra commented 6 years ago

I see that cloud-init has a 721 MiB closure size. That's quite a lot for a glorified version of curl http://169.254.169.254/.

wmertens commented 6 years ago

Indeed, which is why I think it's better to just get the user-data and parse it manually if it's there.

On Wed, Apr 18, 2018 at 3:25 PM Eelco Dolstra notifications@github.com wrote:

I see that cloud-init has a 721 MiB closure size. That's quite a lot for a glorified version of curl http://169.254.169.254/.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/39076#issuecomment-382385364, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWln1gkC_fieJbhPaL2SrMztiwIAzLks5tpz7ZgaJpZM4TYkgY .

wmertens commented 6 years ago

Sorry I see now that this wasn't clear from the issue; part of the conversation was in the PR.

So I think cloud-init is quite bloated and a no-go; we should just get and parse the data ourselves.

So this issue is now about "what data should we support and how do we implement this?".

It would actually be fun to be able to make a solution that wouldn't even require an installation; if we make something that derives a script that is auto-run, then single-shot services could be implemented this way. Quick boot from ISO, bind-mount disk to store, install extra deps and run service.

edolstra commented 6 years ago

Maybe we can factor out and extend the metadata stuff from nixos/modules/virtualisation/amazon-image.nix and nixos/modules/virtualisation/amazon-init.nix into a separate module.

BTW I do have to wonder about the security implications of fetching stuff from 169.254.169.254 in arbitrary environments.

copumpkin commented 6 years ago

Yeah, I'd generally support the idea of expanding the scope of the current "configure from userdata" implementation (I've wanted it on Packet.net, gcloud, etc.), but cloud-init never felt like a good fit to me for NixOS, since it tries to stick a pseudo-declarative interface on top of something that's actually declarative, but there's a lot of bad fits. That's largely why I didn't write a custom NixOS module for cloud-init (apart from it being terribly documented) and instead put the userdata handling natively into a NixOS module.

wmertens commented 6 years ago

@edolstra good point about the security implications - it's a link-local, non-routable address, so if the ISO is booted somewhere and there is a malicious player on the same network and it can fingerprint the OS as NixOS it could conceivably return the correct payload to turn it into a controlled instance while pretending to be a normal server. So perhaps this should go on a separate cloud-installer ISO image instead.

@copumpkin the way I read it, you parse the SSH key from the user data and that's about it, right? I'm rusty on my AWS, can we assume that for every cloud provider that supports cloud-init, we can provide fully custom payload for the user-data?

I'm thinking these might be the steps:

copumpkin commented 6 years ago

@wmertens no, on AWS, you can include an arbitrary NixOS config in userdata and the instance will configure itself to match the specified config at startup time.

wmertens commented 6 years ago

@copumpkin right - but I'd like for us to come up with a system that can be used to configure both ISO and AMI use cases. I mean, in theory you could make a system service that then installs to disk, but more specialized install vs configure attributes would be easier to use and maybe use less memory to evaluate.

Some use cases:

There seem to be a lot of ways that userdata is made available, see http://cloudinit.readthedocs.io/en/latest/topics/datasources.html

All of these sources seem to provide a way to get a Nix configuration text to the VM. Some also provide extra data like network and SSH keys etc. So it seems to me that a Nix build script that is provided with this extra data can then build scripts for installation or configuration, which are then run as part of the bootup.

Then it's up to deciding if it can go in the generic minimal ISO, or if it should be a separate ISO.

I'm certainly not proposing to support all things immediately, my use case is the 169.254.169.254 endpoint with SSH and install.

copumpkin commented 6 years ago

Oh yes, so that's the thing I'm saying I want too.

I just don't really see the benefit of switching over to cloud-init, with all the other baggage it brings with it. I've written custom modules for it, intensely disliked the state of its code and documentation, and adding a list of two or three such datasources never seemed like the end of the world to me. I don't want YAML in my userdata, which it would force me to put there. Its notion of declarative users seems like a bad fit for our notion of it, unless you want to add a big translation layer and get rid of all its attempts to add users, etc.

eliasp commented 6 years ago

Did anyone already look into Ignition?

In general it seems to be designed quite well (having learnt from the issues of cloud-init), comes with a proper specification which is used to validate and provide helpful error context.

Mic92 commented 6 years ago

Ignition does not look like it is compatible with NixOS: https://coreos.com/ignition/docs/latest/examples.html (it modifies configuration itself). I also suspect it makes many assumption about running on CoreOS, that do not apply on other platforms.

copumpkin commented 6 years ago

Yeah, NixOS by default is notch or two above pretty much all declarative config systems, so I'm not sure why we'd take a step back down the declarative ladder to be compatible with something that doesn't map nicely to what we do. We have a "NixOS config in userdata" system today that works and we've been using for years. It's not ideal to have to know the various platform-specific endpoints to hit, but it's not awful either.

ghost commented 4 years ago

Sorry for the noise; just wanted to say this is a really cool idea and I was wondering what the current status of this is.

Mic92 commented 4 years ago

It's not enabled by default, but you could build your own ISOs that have it: https://github.com/nix-community/nixos-generators

wmertens commented 4 years ago

@Mic92 @copumpkin so the conclusion here is: the current system based on cloud-init is good enough for NixOS use, but its closure size is too big for the minimal iso, so it won't get added.

Is that correct?

Mic92 commented 4 years ago

@wmertens Yes. Maybe someone could provide a nix-community project to build variants of NixOS ISOs that we do not have in nixpkgs itself.

flokli commented 4 years ago

Other distros also recognized the cloud-init package and its dependencies is way too much bloat. There is flamingo, meant to be a lightweight replacement for cloud-init in Atomic Host, but is somewhat abandoned.

There's also clearlinux/micro-config, and https://github.com/coreos/coreos-cloudinit.

Some distros seem to be moving to https://github.com/coreos/ignition, which is also a (small) Go binary, but I'm not sure if this can be a replacement, and how well it'd integrate with NixOS.

Mic92 commented 4 years ago

Some distros seem to be moving to coreos/ignition, which is also a (small) Go binary, but I'm not sure if this can be a replacement, and how well it'd integrate with NixOS.

Ignition does not look too bad after grepping through the code. They only reference to /usr/ is /usr/lib/ignition and but can be overwritten exporting SYSTEM_CONFIG_DIR. All executables are looked up through PATH rather than absolute paths...

I think the main issue is that it requires systemd in initrd, which is not what we do right now.

flokli commented 4 years ago

I meant "I'm not sure if this can be a replacement", as Ignition uses another format for its files, and probably we want to keep support for the cloud-init format.

clearlinux/micro-config also doesn't look too complicated to get working.

Mic92 commented 4 years ago

It looks better. One problem I see: Enabling it by default in our minimal iso is a security risk because if an attacker controls the link-local address they can execute arbitrary code. In cloud environment this is fine, but I don't want to expose every user running our installer to that.

flokli commented 4 years ago

Yeah, it probably shouldn't be part of the minimal installer, but it'd be nice to have that tooling available, and emit a NixOS Server|Cloud image with cloud-init support enabled, that could be used by cloud providers.

OddBloke commented 4 years ago

Hey folks, I'm an upstream cloud-init dev and just happened across this thread. I just wanted to respond to one specific comment:

Enabling it by default in our minimal iso is a security risk because if an attacker controls the link-local address they can execute arbitrary code. In cloud environment this is fine, but I don't want to expose every user running our installer to that.

cloud-init has started shipping a systemd generator precisely to handle this case: if it doesn't find that it's running on a cloud platform (by examining DMI data, generally, so this only works on Intel platforms; don't know if that's a concern for NixOS) then it will entirely disable cloud-init for the boot: https://github.com/canonical/cloud-init/blob/master/systemd/cloud-init-generator.tmpl (which relies on https://github.com/canonical/cloud-init/blob/master/tools/ds-identify).

(The version of cloud-init currently in NixOS is too old to include it, but I know that @bcdarwin was looking into updating it earlier today: https://github.com/bcdarwin/nixpkgs/commit/8e126a5ae48e8bd1aa72eba3e29ddcd49535befd)

Mic92 commented 4 years ago

@OddBloke I have updated cloud-init here: https://github.com/NixOS/nixpkgs/pull/95746 If you have time for testing it, we can get it into the next release.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

mbrgm commented 3 years ago

@OddBloke Did you make any progress following @Mic92's PR?

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

msgilligan commented 7 months ago

Some form of cloud-init support (doesn't have to be in minimal image) would allow fast/easy creation of NixOS instances with Lima, see: https://github.com/lima-vm/lima/discussions/430

bam80 commented 7 months ago

Canonical has adopted cloud-init config in it's Ubuntu Server Installer since version 20.04: https://ubuntu.com/server/docs/install/autoinstall

It's not enabled by default, but you could build your own ISOs that have it: https://github.com/nix-community/nixos-generators

If we can't (or don't want, for some unclear to me reason) do the same by default, I would love if we document at least how one could generate the featured installer .iso, if needed.

I'm new to Nix and the cloud-init support in it's installer seems like a very hidden subtle possibility to me. Second day I'm struggling and still have no clear answer if it's possible, at all. While the installing is among the very first things one would do, and shouldn't bring such an obstacles..

nixos-discourse commented 7 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/bluetooth-keyboard-in-stage-1/19259/25

physics-enthusiast commented 7 months ago

Maybe this recent project of mine may be relevant?