search

coreos / ignition

First boot installer and configuration tool
https://coreos.github.io/ignition/
Apache License 2.0
835 stars 245 forks source link

Support disabling initramfs networking via Ignition #979

Closed stbenjam closed 4 years ago

stbenjam commented 4 years ago

Feature Request

Support running part of ignition before networking comes up

Environment

RHCOS

What hardware/cloud provider/hypervisor is being used to run Ignition?

Baremetal

Desired Feature

For baremetal IPI platform in OpenShift, we load ignition from disk. We would like to include network configuration that happens before NetworkManager starts. For example, we want to disable dhcp on some interfaces. If we can't do that, then on some hosts that could have upwards of 10 (!) interfaces, booting takes an extraordinary long time.

Also, when an ignition contains an 'append' with a network source, it's conceivable we need to do some pre-configuration to make that network endpoint accessible (for example, we need to configure a VLAN on an interface).

Other Information

It's almost like we need a pre-ignition tool (preflight?) analogous to what afterburn is doing post-ignition.

stbenjam commented 4 years ago

@cgwalters What do you think of something like this?

hardys commented 4 years ago

To link this to a real (OpenShift) use-case, see https://bugzilla.redhat.com/show_bug.cgi?id=1824331 - in some baremetal environments the external network can be on a tagged vlan, so you have to do some configuration via ignition before the append (in v2.x) URL can be evaluated. I guess a similar issue exists with the merge feature in v3.

cgwalters commented 4 years ago

A lot of overlap here with https://github.com/openshift/enhancements/pull/291

cgwalters commented 4 years ago

See also https://github.com/coreos/ignition-dracut/issues/94

stbenjam commented 4 years ago

A lot of overlap here with openshift/enhancements#291

I did read through that but it's very tied to solving the problem by embedding an ignition in an ISO, rather than letting ignition do some things before and after networkmanger is up.

stbenjam commented 4 years ago

See also coreos/ignition-dracut#94

Yea that's a similar problem, but we are using https://coreos.com/os/docs/latest/config-drive.html instead, so I'm not sure this issue helps us

cgwalters commented 4 years ago

What I'd argued in one of the epic number of tickets around this is that I think all we need to do for all the major clouds that use the link-local address is bring up networking enough to do that, then we could pull network configuration out of the Ignition itself - then we can do static networking in clouds too (not that most people want this).

That said, if we fixed https://github.com/openshift/machine-config-operator/issues/1690 then you guys could "flatten" the config into /boot/config.ign and then you wouldn't need networking in the initramfs at all, right? I go back and forth on whether or not this is a good idea; it takes away some control from the MCO.

rather than letting ignition do some things before and after networkmanger is up.

But we'd need to flesh out more what that would look like. Something like:

"initramfs-network": {
    "dhcp": false,
    "args": "ip=192.xxx...."

Or just support writing arbitrary things into /etc/NetworkManager before it starts, basically a filesystem bit that applies to the initramfs?

stbenjam commented 4 years ago

That said, if we fixed openshift/machine-config-operator#1690 then you guys could "flatten" the config into /boot/config.ign and then you wouldn't need networking in the initramfs at all, right? I go back and forth on whether or not this is a good idea; it takes away some control from the MCO.

That addresses the immediate problem for us, but not the general use. It makes complete sense to use network data sources for append/merge in ignition, but there's always going to be a circular dependency on environments that have networks that need configuration.

rather than letting ignition do some things before and after networkmanger is up. But we'd need to flesh out more what that would look like. Something like: "initramfs-network": { "dhcp": false, "args": "ip=192.xxx...." Or just support writing arbitrary things into /etc/NetworkManager before it > starts, basically a filesystem bit that applies to the initramfs?

Maybe? I don't understand entirely how the initramfs dracut/ignition stuff interact -- is the /etc/NetworkManager at that stage in the ramdisk or is that the host's eventual configuration?

cgwalters commented 4 years ago

We may need a meeting on this but I really want to walk though how the Live ISO changes the game for you guys. Hmm....I think we maybe do need to support some flag/mechanism for the Live ISO to disable networking in the initramfs even if Ignition is injected.

Maybe? I don't understand entirely how the initramfs dracut/ignition stuff interact -- is the /etc/NetworkManager at that stage in the ramdisk or is that the host's eventual configuration?

We use NM in the initrd too (RHCOS will after 8.2). There's no required linkage between the initramfs configuration and the real root; most people will want them to be the same (see https://github.com/coreos/ignition-dracut/pull/89 ) but like that PR we can add an option to avoid propagating them to the real root.

jlebon commented 4 years ago

Related: https://github.com/coreos/ignition/pull/956 (which would also be smart enough to handle live ISO with embedded Ignition configs). It's also an on-ramp for supporting fetching Ignition configs over link-local addresses.

jlebon commented 4 years ago

I'm very hesitant to allow Ignition to configure the initramfs itself. I'd prefer we stay away from that for as long as we can because it essentially introduces the same problem for the initrd which Ignition was meant to solve for the real root.

It makes complete sense to use network data sources for append/merge in ignition, but there's always going to be a circular dependency on environments that have networks that need configuration.

We've been working hard on improving the situation there. :) In addition to the links above, see also https://github.com/coreos/fedora-coreos-tracker/issues/460. I agree with Colin that we should push further on the live ISO embedded config path for bare metal, and investigate the link-local approach for clouds. (I'm also not sure how Ignition can correctly implement the RFE here if e.g. you configure networking in the base config, but then one of the merge configs actually wants to override what the config file should have been.)

stbenjam commented 4 years ago

I'm very hesitant to allow Ignition to configure the initramfs itself. I'd prefer we stay away from that for as long as we can because it essentially introduces the same problem for the initrd which Ignition was meant to solve for the real root.

Would it be kernel command line arguments then for this configuration? I wasn't sure if that worked in rhcos, since we tried it when we were trying to get IPv6 working. Can you do VLAN configuration with those arguments?

It makes complete sense to use network data sources for append/merge in ignition, but there's always going to be a circular dependency on environments that have networks that need configuration. We've been working hard on improving the situation there. :) In addition to the links above, see also coreos/fedora-coreos-tracker#460. I agree with Colin that we should push further on the live ISO embedded config path for bare metal, and investigate the link-local approach for clouds.

The ISO approach for bare metal isn't going to work in the IPI case, and that seems to be ignored in the present designs. We do exist and have customers. We have some initial discussions about how we can use the live ISO for IPI but it's going to be later than we need this problem solved.

I also don't see how the live ISO case fits into OpenShift, because the installer generates a stub ignition with network dependencies and you're going to run into customers with these complex network requirements. I don't see where that use case is being addressed.

(I'm also not sure how Ignition can correctly implement the RFE here if e.g. you configure networking in the base config, but then one of the merge configs actually wants to override what the config file should have been.)

You could have something like afterburn to be able to do preflight configuration before ignition runs.

OpenStack config drives have network metadata for similar reasons: https://docs.openstack.org/nova/latest/user/metadata.html.

cgwalters commented 4 years ago

I'm very hesitant to allow Ignition to configure the initramfs itself. I'd prefer we stay away from that for as long as we can because it essentially introduces the same problem for the initrd which Ignition was meant to solve for the real root.

If it's just constrained to networking though, that seems not too bad.

And actually perhaps to start, we could offer exactly one thing - the ability for Ignition to disable initramfs networking. That would allow using coreos-installer iso embed to be fully programmable - the system boots live, the live Ignition can bring up the network however it wants (or not), and run coreos-installer however it wants, providing that Ignition config however it wants, etc.

In particular once we have osmet, I suspect a lot of cases could get away with not even bringing up the network at all in the live OS.

cgwalters commented 4 years ago

Yeah I am increasingly certain that we missed this important case of "Ignition provided, but does not require networking in the initramfs" in all of the recent work following https://github.com/coreos/fedora-coreos-tracker/issues/443

I took a look at the recent libvirt testing scripts and it looks like to me theyl use the default bridged network which still has DHCP enabled. So what happens here is when we're doing a static config in those tests...we still used (and required!) DHCP in the initramfs.

That's not going to work for environments without DHCP at all, or even just ones where we don't want to DHCP on each NIC but not all of them etc.

https://github.com/openshift/enhancements/pull/291 is not complete IMO until we fix this.

Oh right actually, this is exactly what this issue is: https://github.com/coreos/coreos-installer/issues/164#

cgwalters commented 4 years ago

OK so yep, going back we have two "not so complex" options:

1) coreos-install iso embed --no-initramfs-networking: https://github.com/coreos/coreos-installer/issues/164#issue-567060872 (only applies to live ISO) 2) Enhance Ignition to have a way for the config to tell us not to use the network at all in the initramfs (this issue)

Or, the much bigger options of:

3) Enhance Ignition to have a way to configure initramfs networking 4) Enhance Ignition to do conditional networking

Another approach is:

5) Don't enable networking in the initramfs in the Live ISO at all (but that seems very confusing and breaks the symmetry)

dustymabe commented 4 years ago

Yeah I am increasingly certain that we missed this important case of "Ignition provided, but does not require networking in the initramfs" in all of the recent work following coreos/fedora-coreos-tracker#443

A follow-on that we still want to do is https://github.com/coreos/fedora-coreos-tracker/issues/460

I took a look at the recent libvirt testing scripts and it looks like to me theyl use the default bridged network which still has DHCP enabled. So what happens here is when we're doing a static config in those tests...we still used (and required!) DHCP in the initramfs.

Can you elaborate what you mean. That test script tests many different configurations including dhcp and static networking in the initramfs combined with static networking in the real root. I do use the default libvirt bridged network, but that's only to cover the dhcp test cases and the machine won't attempt DHCP if you tell it not to.

That's not going to work for environments without DHCP at all, or even just ones where we don't want to DHCP on each NIC but not all of them etc.

The EPIC work that I've done specifically focuses on static networking via the installer in the Live ISO. In that case you can easily configure static networking when doing the install.

Follow on work to this is to not bring up networking in the initramfs if not needed at all: https://github.com/coreos/fedora-coreos-tracker/issues/460, but that was not part of the EPIC.

openshift/enhancements#291 is not complete IMO until we fix this.

Oh right actually, this is exactly what this issue is: coreos/coreos-installer#164

I think the better approach is https://github.com/coreos/fedora-coreos-tracker/issues/460 . For now (RHCOS in 4.6) you can achieve 'no networking in the initramfs' like:

but I guess the real question is why do you need "no networking in the initramfs". I think the real desire is "I only want the networking I want in the initramfs". In that case just configure the networking you want via the installer right?

jlebon commented 4 years ago

Heh, was typing up a comment but @dustymabe beat me to it. :)

To summarize: In the general case, I think this is resolved by https://github.com/coreos/fedora-coreos-tracker/issues/460 (which provides a framework for apps to tell the OS that networking is needed) and https://github.com/coreos/ignition/pull/956 + https://github.com/coreos/ignition-dracut/pull/164/ (which makes use of that framework). In the live ISO case/metal case of "complex networking on first boot", one can configure networking as required at install time and use --copy-network.

dustymabe commented 4 years ago

@jlebon yes. Maybe we should all get together and write up a proper EPIC so the requirements are collected and we can make sure https://github.com/coreos/fedora-coreos-tracker/issues/460 addresses all needs.

cgwalters commented 4 years ago

provide --firstboot-args='' with kargs that disables networking in the initramfs and provide networking config in the real root.

The user story here is "I have a CoreOS ISO and coreos-installer iso embed" and I want to automate installs. --firstboot-args is something that runs in the booted OS. I want to be able to e.g. run a script before or after coreos-install without having ever done DHCP at all.

cgwalters commented 4 years ago

To summarize: In the general case, I think this is resolved by coreos/fedora-coreos-tracker#460 (which provides a framework for apps to tell the OS that networking is needed) and #956 + coreos/ignition-dracut#164 (which makes use of that framework).

Right, I added that as a second option in the "more complex" options above.

dustymabe commented 4 years ago

The user story here is "I have a CoreOS ISO and coreos-installer iso embed" and I want to automate installs. --firstboot-args is something that runs in the booted OS. I want to be able to e.g. run a script before or after coreos-install without having ever done DHCP at all.

Yep. We did have https://github.com/coreos/fedora-coreos-config/pull/326 to not require networking for the ISO but we explicitly do bring up networking in the case of an embedded ignition config (as requested during feature discussions) and pointing to future work to take care of that case: https://github.com/coreos/fedora-coreos-config/blob/5615f58487f9fd9f1ebc19e0e6416b53a3e6270f/overlay.d/05core/usr/lib/dracut/modules.d/20live/coreos-liveiso-network-kargs.service#L5-L15

We could easily remove the ConditionPathExists=|/config.ign

cgwalters commented 4 years ago

and pointing to future work to take care of that case:

"but the user can override with rd.neednet=0 now if needed."

No, there's no ergonomic way to override the live ISO kernel arguments in an automated fashion.

We could easily remove the ConditionPathExists=|/config.ign

That's an option I guess...I added it to the list above.

jlebon commented 4 years ago

I want to be able to e.g. run a script before or after coreos-install without having ever done DHCP at all.

Right yeah, this is also in scope for https://github.com/coreos/fedora-coreos-tracker/issues/460. Right now we took the shortcut that embedded config = requires networking. But the end goal is absolutely for Ignition to only ask for networking if required (via the exact same mechanism that is used in clouds/freshly installed metal).

One gap then is if one needs complex networking on first boot of a non-metal platform, and that's what Afterburn is trying to address (by providing a channel for configuring networking, see e.g. https://github.com/coreos/afterburn/pull/404).

Of course, another major gap for much of this is to have osmet supported on RHCOS so that install-time can truly be fully offline. We should chat about our approach on that! (And I see now you recently opened https://github.com/coreos/coreos-assembler/pull/1467 related to this).

stbenjam commented 4 years ago

This discussion seems to be going off the rails a bit from our original feature request. I'm not sure why the live CD is part of this particular discussion, we're not using it -- maybe we should have a live discussion about what ignition is missing for us? We need to have something in the 4.6 time frame.

OpenShift installer creates an ignition that needs networking. Any ignition has the possibility to include merge/appends that reference network URI's -- what is the answer when a user requires some configuration to make networking work to access those resources?

It's impossible today in the baremetal IPI platform to support use cases where the network used to access the MCS requires a VLAN, for example.

dustymabe commented 4 years ago

From https://github.com/coreos/ignition/issues/979#issuecomment-632074230

Another approach is:

Don't enable networking in the initramfs in the Live ISO at all (but that seems very confusing and breaks the symmetry)

Networking would still be enabled in the other conditions. Just not for the embedded ignition config case.

jlebon commented 4 years ago

maybe we should have a live discussion about what ignition is missing for us?

:+1: Yeah, that'd be great! :)

The reason we're talking about the live ISO is because it will break the dependency loop between Ignition and networking by being able to provide networking configuration upfront at install time.

ashcrow commented 4 years ago

maybe we should have a live discussion about what ignition is missing for us?

+1 Yeah, that'd be great! :)

Agreed. The result of this spike should find it's way into a card for execution as well once we've figured out what should be done.

cgwalters commented 4 years ago

This discussion seems to be going off the rails a bit from our original feature request. I'm not sure why the live CD is part of this particular discussion, we're not using it -- maybe we should have a live discussion about what ignition is missing for us?

Yes, I understand. If we want to handle your case of using the -openstack image (right?) then we can't use option 1.

OpenShift installer creates an ignition that needs networking. Any ignition has the possibility to include merge/appends that reference network URI's -- what is the answer when a user requires some configuration to make networking work to access those resources?

OK I thought about this more and you're right, we either need to:

  1. Make it easy to get a "flattened" real config https://github.com/openshift/machine-config-operator/issues/1690
  2. Support configuring the initramfs network via Ignition

Of these...I lean more towards the first, but it's hard. The reasons (AFAIK) openshift-install always generates a pointer configuration is twofold:

Anyways, once we have a flattened config, then as long as we have a way to just disable the initramfs networking requirement, we only have to concern ourselves with networking in the real root.

stbenjam commented 4 years ago

This discussion seems to be going off the rails a bit from our original feature request. I'm not sure why the live CD is part of this particular discussion, we're not using it -- maybe we should have a live discussion about what ignition is missing for us?

Yes, I understand. If we want to handle your case of using the -openstack image (right?) then we can't use option 1.

That's what we use today, I'd expect us to be able to move the IPI install for baremetal to using the live RHCOS image eventually, but not in 4.6.

OpenShift installer creates an ignition that needs networking. Any ignition has the possibility to include merge/appends that reference network URI's -- what is the answer when a user requires some configuration to make networking work to access those resources?

OK I thought about this more and you're right, we either need to:

1. Make it easy to get a "flattened" real config [openshift/machine-config-operator#1690](https://github.com/openshift/machine-config-operator/issues/1690)

2. Support configuring the initramfs network via Ignition

Of these...I lean more towards the first, but it's hard. The reasons (AFAIK) openshift-install always generates a pointer configuration is twofold:

* In e.g. AWS, there are size limitations on user data, but I suspect you don't have that problem

We store the user data in a secret, so we hit the 1MB limitation from Kuberenetes. Ironic's own limitation is 64MB. 1MB is maybe a bit small but it's reasonable for the general case.

* The actual Ignition lives inside the MCO codebase and the installer binary just doesn't have it "offline".   Solving that would be...hard and messy; the MCO (most notably the MCS) is just designed to run as part of a cluster.  Really we need something like a bootstrap node to extract that configuration.

Anyways, once we have a flattened config, then as long as we have a way to just disable the initramfs networking requirement, we only have to concern ourselves with networking in the real root.

This addresses our immediate need. For the installer, I already wrote up a simple POC that flattens an ignition with terraform: https://github.com/openshift-metal3/terraform-provider-openshift/blob/master/openshift/data_source_inline_ignition_v22.go#L35. We can make that work somehow, we just need a way to access the MCS in the cluster to do the same for workers.

We run into edge cases where users may do things that end up exceeding 1MB for the flattened ignition, which is why the second option of having network support for the initramfs is appealing to me. It also handles the very generalized case for anyone using ignition. Presumably there's some future possibility that ignition ends up in RHEL, or CoreOS is used more generally and we need to think about that.

cgwalters commented 4 years ago

OK some members of the CoreOS team had a live chat about this issue (and we're going to schedule a followup meeting with you guys) but here are some conclusions:

1) There is general agreement that our epics around the Live ISO and static networking are not complete until we have a way to disable the initramfs networking requirement for the live ISO boot when Ignition is provided 2) We will not add any changes to the OpenStack image or Ignition in general to the epic at this time for 4.6 3) There is general agreement that the "conditional networking" change for Ignition is the most elegant fix (among others it will speed up cosa run w/qemu)

I think a corollary to this is we'd like to have a discussion about how hard it really is for you guys to switch to the Live ISO.

stbenjam commented 4 years ago

That sounds great, it'd be good to invite @hardys, @kirankt and @juliakreger as well.

cgwalters commented 4 years ago

https://github.com/coreos/fedora-coreos-config/pull/415

cgwalters commented 4 years ago

Schedule intersections are hard, sent something for 2pm EST next Tuesday, hope that works.

kirankt commented 4 years ago

This discussion seems to be going off the rails a bit from our original feature request. I'm not sure why the live CD is part of this particular discussion, we're not using it -- maybe we should have a live discussion about what ignition is missing for us?

Yes, I understand. If we want to handle your case of using the -openstack image (right?) then we can't use option 1.

OpenShift installer creates an ignition that needs networking. Any ignition has the possibility to include merge/appends that reference network URI's -- what is the answer when a user requires some configuration to make networking work to access those resources?

OK I thought about this more and you're right, we either need to:

1. Make it easy to get a "flattened" real config [openshift/machine-config-operator#1690](https://github.com/openshift/machine-config-operator/issues/1690)

Just wanted to add my 2 cents for this Option 1. We will need this for Day 2 use as well. Once the install is finished, we will still need to get to flattened ignition from Metal3 ecosystem for provisioning additional nodes.

2. Support configuring the initramfs network via Ignition

Of these...I lean more towards the first, but it's hard. The reasons (AFAIK) openshift-install always generates a pointer configuration is twofold:

* In e.g. AWS, there are size limitations on user data, but I suspect you don't have that problem

* The actual Ignition lives inside the MCO codebase and the installer binary just doesn't have it "offline".   Solving that would be...hard and messy; the MCO (most notably the MCS) is just designed to run as part of a cluster.  Really we need something like a bootstrap node to extract that configuration.

Anyways, once we have a flattened config, then as long as we have a way to just disable the initramfs networking requirement, we only have to concern ourselves with networking in the real root.

miabbott commented 4 years ago

Folks from the KNI team and CoreOS team had a meeting today about this issue. Notes captured here - https://hackmd.io/QqY2xZGVSVqO8L8zBPqM7w

cgwalters commented 4 years ago

The TL;DR on this is that for OpenShift <= 4.6 at least we will recommend that anyone who wants to control the initramfs networking for anything except the Live ISO to mount the disk image and edit the kernel cmdline.

We already support networking on the cmdline for PXE cases, so this is fully symmetric.

Future work will focus on work mostly already outlined in https://github.com/openshift/enhancements/pull/291

hardys commented 4 years ago

@cgwalters Just wanted to see if we can reach a conclusion on the next steps here, as I'm not clear that we've got a plan to resolve both use-cases in the original issue.

  1. Ability to disable networking (or specific interfaces) in the ramdisk - this will be handled via cmdline args (which isn't ideal for IPI as we have one PXE config for all nodes...), and in future this may be improved by openshift/enhancements#291

  2. How to retrieve the rendered ignition config from the MCS, when the network it needs is on a non-default VLAN. It's not clear to me we have any solution for this case yet, and it will potentially be a common requirement for baremetal environments, can you help me understand the next steps here please? Thanks!

cgwalters commented 4 years ago

First I want to be clear - you are specifically requesting this for the OpenStack image right? In other words this is a request for a targeted "baremetal IPI <= 4.5 quick patch" and not some sort of general change, right? Because I think the conclusion here is our "general change" is the Live ISO + vmware backchannels already elaborated in the enhancement you linked.

Ability to disable networking (or specific interfaces) in the ramdisk

Isn't that covered by https://github.com/coreos/ignition/issues/979#issuecomment-634800899 ? i.e. "open up and edit the disk" for versions <= 4.5 ?

How to retrieve the rendered ignition config from the MCS, when the network it needs is on a non-default VLAN. It's not clear to me we have any solution for this case yet, and it will potentially be a common requirement for baremetal environments, can you help me understand the next steps here please? Thanks!

If we definitively conclude we need that, let's discuss in https://github.com/openshift/machine-config-operator/issues/1690

ashcrow commented 4 years ago

cc'ing @imcleod for visibility as this is a feature request

hardys commented 4 years ago

First I want to be clear - you are specifically requesting this for the OpenStack image right? In other words this is a request for a targeted "baremetal IPI <= 4.5 quick patch" and not some sort of general change, right? Because I think the conclusion here is our "general change" is the Live ISO + vmware backchannels already elaborated in the enhancement you linked.

We tried the quick-hack for baremetal and it got complex (ref https://github.com/openshift/machine-config-operator/issues/1690 as one issue we encountered) so ideally I think we should try to solve this more generally than some special solution that only works for IPI baremetal?

I don't actually think this necessarily has to be backported or work with older releases, the goal is to provide a supportable interface in some future release.

Ability to disable networking (or specific interfaces) in the ramdisk

Isn't that covered by #979 (comment) ? i.e. "open up and edit the disk" for versions <= 4.5 ?

This approach was rejected by $customer but if it's documented somewhere as a supported approach it may work as an interim solution. The main issue I can see is you may end up needing multiple custom images if you have pools of nodes with different nic setups.

How to retrieve the rendered ignition config from the MCS, when the network it needs is on a non-default VLAN. It's not clear to me we have any solution for this case yet, and it will potentially be a common requirement for baremetal environments, can you help me understand the next steps here please? Thanks!

If we definitively conclude we need that, let's discuss in openshift/machine-config-operator#1690

I think this is a fairly common requirement in the case where you want to PXE boot machines, then have the controlplane traffic on a tagged network attached to the same NIC. I guess this same issue exists for UPI deployments in that case.

If it helps I'm happy to close this and work on an enhancement (or track via https://github.com/openshift/machine-config-operator/issues/1690), this issue was raised I think with the idea that we wanted ignition to grow some "early config" interface, but it seems like it may be preferable to focus on making it easier to inject the combination of the pointer config and the fully rendered one?

It seems like that would also be required in the ISO embed solution, as you still need to consume the combination of the pointer config (which the user might append config to using the create ignition-configs interface of openshift-install) and the rendered config?

cgwalters commented 4 years ago

so ideally I think we should try to solve this more generally than some special solution that only works for IPI baremetal?

We believe that the general solution is the Live ISO.

hardys commented 4 years ago

so ideally I think we should try to solve this more generally than some special solution that only works for IPI baremetal?

We believe that the general solution is the Live ISO.

Understood - How do you envisage this working in the PXE boot case?

cgwalters commented 4 years ago

Understood - How do you envisage this working in the PXE boot case?

Via the ip= kernel argument - same way this has been done with PXE for a long time, right?

cgwalters commented 4 years ago

It seems like that would also be required in the ISO embed solution, as you still need to consume the combination of the pointer config (which the user might append config to using the create ignition-configs interface of openshift-install) and the rendered config?

No; the Ignition provided to the ISO is just a wrapper around running coreos-installer to a target disk plus providing the pointer Ignition, and possibly including any pre-defined networking configuration.

On the second boot, the OpenShift pointer ignition will be used alongside the pre-injected network configuration.

hardys commented 4 years ago

Via the ip= kernel argument - same way this has been done with PXE for a long time, right?

That works when you boot the ramdisk that writes RHCOS to disk, but what about the first boot of RHCOS after that, when it has only the pointer ignition but also requires some network configuration before it can resolve the pointer URL to download the full config?

On the second boot, the OpenShift pointer ignition will be used alongside the pre-injected network configuration.

It sounds like I'm missing something fundamental here - how is that pre-injected network configuration provided to RHCOS for the first boot, so it can resolve the pointer ignition, by customizing the deployed RHCOS disk image, or is there some other method?

That is the crux of the problem we're trying to solve, and I don't think it is dependent on how the RHCOS image gets written to disk, on first boot it still has to download the config via the pointer ignition?

Thanks for the help figuring this out! :)

jlebon commented 4 years ago

On the second boot, the OpenShift pointer ignition will be used alongside the pre-injected network configuration.

It sounds like I'm missing something fundamental here - how is that pre-injected network configuration provided to RHCOS for the first boot, so it can resolve the pointer ignition, by customizing the deployed RHCOS disk image, or is there some other method?

See https://github.com/coreos/coreos-installer/pull/212 and https://github.com/coreos/fedora-coreos-config/pull/346. IOW, at install-time, you pass --copy-network to coreos-installer install and that will inject the NM keyfiles into /boot. Those keyfiles are then copied into the initramfs at first boot time before Ignition needs to fetch anything.

So in essence, your live ISO/PXE Ignition config would include three things:

  1. the "base" Ignition config of the target install
  2. any NM keyfiles needed for custom networking (note that in the live ISO boot, Ignition won't try to turn on networking in the initrd, so you can use it properly to configure the networking of the live real root directly if that's needed to e.g. fetch the RHCOS metal image or Ignition config)
  3. a systemd service that runs coreos-installer install /dev/foo --ignition base.ign --copy-network ...
hardys commented 4 years ago

See coreos/coreos-installer#212 and coreos/fedora-coreos-config#346. IOW, at install-time, you pass --copy-network to coreos-installer install and that will inject the NM keyfiles into /boot. Those keyfiles are then copied into the initramfs at first boot time before Ignition needs to fetch anything.

Thanks @jlebon - this is the detail I was missing :)

I wonder if we could consider making the 15copy-installer-network script copy from a config-drive partition if it exists? One of the problems when the ironic deploy ramdisk is used (as is currently the case for IPI baremetal) it can write a config-drive partition, but there's no support for modifying the OS disk image itself (so we can't drop files in /boot).

Long term we can also look at how we might consume the rhcos initrd with coreos-install, but there are some feature gaps and interfaces differences which will need further investigation if we want that to work with the ironic services we currently use.

jlebon commented 4 years ago

I wonder if we could consider making the 15copy-installer-network script copy from a config-drive partition if it exists?

Hmm... maybe. (Though the fact that it's optional makes it similar to https://github.com/coreos/ignition/issues/928.) Let's see if there's a better way first.

One of the problems when the ironic deploy ramdisk is used (as is currently the case for IPI baremetal) it can write a config-drive partition, but there's no support for modifying the OS disk image itself (so we can't drop files in /boot).

Do you have a link to documentation/code re. what the deployer is capable of? I presume you're not able to run arbitrary code?

Long term we can also look at how we might consume the rhcos initrd with coreos-install, but there are some feature gaps and interfaces differences which will need further investigation if we want that to work with the ironic services we currently use.

Yes, agreed! :+1:

dustymabe commented 4 years ago

Long term we can also look at how we might consume the rhcos initrd with coreos-install, but there are some feature gaps and interfaces differences which will need further investigation if we want that to work with the ironic services we currently use.

Long term I think there are a few options with varying levels of difficulty so we might progressively go through them:

  1. teach the deployer how to mount /boot from the disk that was recently imaged and modify contents
    • This is the easiest to implement, but not ideal as the deployer is now responsible for knowing internal details of how we are moving things around in RHCOS/FCOS land.
  2. include the coreos-installer binary in the deployer and have the deployer call coreos-installer with specified options to image the disk
  3. have the deployer actually be the RHCOS/FCOS live PXE/ISO environment in the future.
hardys commented 4 years ago

One of the problems when the ironic deploy ramdisk is used (as is currently the case for IPI baremetal) it can write a config-drive partition, but there's no support for modifying the OS disk image itself (so we can't drop files in /boot).

Do you have a link to documentation/code re. what the deployer is capable of? I presume you're not able to run arbitrary code?

There is an overview here https://docs.openstack.org/ironic/latest/user/index.html#understanding-bare-metal-deployment - the agent itself is https://github.com/openstack/ironic-python-agent which is contained in a ramdisk we use to write the OCP disk image (similar to the OCP ramdisk with coreos-install).

Basically there are no interfaces for running arbitrary code, or injecting arbitrary files, the baremetal box is treated like a cloud VM - you provide an OS disk image, and optionally some user-data/meta-data (which in the metal3 case happens via a config-drive partition)

OpenStack has support for a network_data.json ref https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata but I don't think the ignition openstack backend does anything with that atm?