Support default kernel arguments

[vtolstov]

How can i add some things to kernel cmdline when compose tree? Or how i control cmdline when boot ostree system?

[cgwalters]

ostree admin deploy --karg=. Which is used by e.g. the Fedora/CentOS Anaconda installer

[vtolstov]

Thanks, i miss it with using rpm-ostree =(. This is really hard use rpm-ostree without ostree admin =). But not ostree admin upgrade works fine, because i'm deploy ref via ostree admin deploy and it not complain about unknown refspec

[vtolstov]

Last qeustion - i'm add custom cmdline with treefile (dracut additional cmdline and specify kernel cmdline to dracut). Why after composing tree i don't have this options in dracut config and grub entries does not contains this line?

[cgwalters]

We don't currently support that, but we could.

[cgwalters]

Basically for now though, use the kickstart https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst#bootloader verb.

[paulvt]

Another thing to note is that --karg allows one to update or set a kernel boot argument, but not to remove one. For know, I use --karg=foo= to at least clear it, but the relevant part of the kernel does complain it cannot handle the value ''.

[dustymabe]

Marking this as medium difficulty, although I suspect it might lean towards hard. @cgwalters WDYT?

[cgwalters]

Marking this as medium difficulty, although I suspect it might lean towards hard. @cgwalters WDYT?

As a general rule I think for ostree/rpm-ostree, the difficulty level is weighted to a much greater degree by the person's knowledge (e.g. bootloaders, C or Rust, whether it requires learning the test framework etc.) compared to other projects. I understand the goal of labeling the difficulty level of issues, but I'm not sure it's worth trying to figure out the exact levels here. If someone wants to dive in they should feel free to ask questions on an issue I'd say.

[dustymabe]

Yep. It's hard to get a standard metric. I'm just trying to get a ballpark estimate so that we can get an idea and group issues into buckets. Maybe there will be some easier medium tasks and some harder medium tasks, but that's probably ok.

Since we have ~150 open issues, if we have buckets of good-first-issue and difficulty/medium tasks then we have a starting point and also a ramp up for new contributors.

If you'd like me to stop this endeavor I will.

[rfairley]

Just to give a little input, I do like the good-first-issue and difficulty/medium split to roughly differentiate the size of the change. good-first-issue usually means a 1-10 line change, mostly involving rearranging existing logic and possibly adding a test, and difficulty/medium might be adding something new but still only touches one or two components. But I can see how the difficulty/medium would get easier once one is familiar with the codebase, so the size of the change doesn't necessarily indicate difficulty.

I find the difficulty/medium label is still handy for quickly looking up issues that should be straightforward but more involved than the good-first-issue type - as a "next step" once one has done a good-first-issue and is more familiar with the codebase.

I agree that asking is the best way to get an idea of the difficulty and if one has the specific prereq. knowledge for it. difficulty/medium might just be a way to group those tasks that one might consider doing after completing a good-first-issue.

[cgwalters]

I have absolutely no problem with you or anyone trying to classify the issues! I just feel like for me personally the benefit/time ratio is too low to worry too much about whether an issue is medium/high or whatever.

[dustymabe]

+1

I'll stop asking for input, but will reserve the right to be slightly or significantly wrong in my estimates :)

[rfairley]

Some thoughts after reading over the code around kargs:

We have mentioned having OSTree commits contain the default kargs. Could this take the form of a file containing the default kargs which gets committed to the ostree, e.g. /usr/lib/ostree/kargs? OSTree would read the default kargs from that file during a deployment, and append the kargs to the state file at "/run/ostree/staged-deployment", and the BLS fragment on reboot.

For rpm-ostree treefiles, I'm thinking we could add a field e.g. default-kargs or just kargs, where the user specifies a list of key-value pairs for each karg (arg, value). The key-value pairs would be written to /usr/lib/ostree/kargs at compose time, and that file would be committed to the ostree.

This still means mangling the defaults with any kargs specified with ostree admin deploy --karg= or rpm-ostree kargs - I'm not sure of how to keep those separate other than storing them as suggested previously. There are other ways we could make the mangling more visible, e.g. a switch to print out the default, user-specified, and mangled kargs during ostree admin --karg= during the deploy, or have additional fields containing the default and user kargs to the state file in /run.

If the user wanted to completely override the default kargs, we could also support /etc/ostree/kargs which would shadow /usr/lib/ostree/kargs.

How does the file idea sound?

[cgwalters]

Maybe /usr/lib/ostree-boot/kargs? Today that gets copied into /boot but we could teach the library to skip that one file?

Another bikeshed issue I see here is: what is the file format of /usr/lib/ostree/kargs? Newline-separated plain text? JSON? Do we need to care about kargs with non-UTF-8 data?

If the user wanted to completely override the default kargs, we could also support /etc/ostree/kargs which would shadow /usr/lib/ostree/kargs.

Well this gets tricky since today one can configure kernel args, they live the deployment's BLS fragments in /boot. But that's the "merged" state, so it mixes user-supplied and system default which we want to get away from. So probably indeed /etc/ostree/kargs would make sense.

There's a very special hazard with this though which is trying to do an upgrade from an existing commit to one which uses this - if the "source of truth" of kargs switches to /{/usr,/etc}/ostree/kargs then we would drop any kargs which the user explicitly configured in their previous deployment.

Maybe what we need to do is add something like:

# ostree: generated from /usr/lib/ostree/kargs and /etc/ostree/kargs
options rd.lvm.lv=sb/root root=/dev/mapper/sb-root ostree=/ostree/boot.0/fedora-silverblue/78f3e4c34baaa9af28885bb3b6d92db7e59b73a9261d68a566de1c66230534dc/0 coreos.oem.id=qemu

And that comment acts as a marker that says "use deployment for source of truth on upgrades, not the merge deployment's kargs"?

[rfairley]

Maybe /usr/lib/ostree-boot/kargs? Today that gets copied into /boot but we could teach the library to skip that one file?

Sounds good to me! Makes sense to keep the files relating to boot in ostree-boot where it'll be seen close to the other boot files.

what is the file format of /usr/lib/ostree/kargs? Newline-separated plain text? JSON? Do we need to care about kargs with non-UTF-8 data?

Had been wondering between those formats, was also thinking the kargs file could be a keyfile (to the Desktop Entry Specification). I think during parsing to a GKeyFile this would help enforce a definite "key-value pairs only, one per line" format. Though validating JSON could accomplish that too, and it'd be clean to work with translating from the treefile.

Is the the encoding of the kernel parameters dependent on the locale of the system, or is there software that might specially need non-UTF-8 in the kargs? If we did support non-UTF-8, would plain text be a better option (with some switch to specify the encoding of the kargs)?

# ostree: generated from /usr/lib/ostree/kargs and /etc/ostree/kargs

If we had this comment be a marker, would libostree delete that comment and apply the new kargs if the user did rpm-ostree kargs or ostree admin deploy --kargs=? Then with the comment not present, the merge deployment becomes the source of truth (like it is today). If the comment is present, then OSTree would regenerate from the deployment (i.e. /usr/lib/ostree/kargs and /etc/ostree/kargs).

To reset the source of truth, we could also have a command like rpm-ostree kargs --reset or rpm-ostree kargs --regenerate which would rewrite the BLS config from /usr/lib/ostree/kargs and /etc/ostree/kargs.

[rfairley]

Just another question, which isn't blocking the initial patch I'm working on, but came up while thinking about the design:

Should the host config (/etc/ostree/kargs) have an equivalent of rpm-ostree kargs --delete when merging/overriding with the base config (/usr/lib/ostree-boot/kargs)? I'm not sure of a specific need for this, but I wonder if a user might want full control of the kargs through their host config to override and delete kargs in the base config, while using the etc/usr configs as a source of truth. That way the host would still be "subscribed" to updates to the /usr/lib/ostree-boot/kargs file in newer ostrees. We could also support the --append behavior easily as well if we did this.

I imagine /etc/ostree/kargs would need a format like the following to support that:

d root=FOO # d <=> --delete
d KEY=VAL
r rd.lvm.lv=sb/root # r <=> --replace
a coreos.oem.id=qemu # a <=> --append
...

or

[kargs]
delete="root=FOO KEY=VAL ..."
replace="rd.lvm.lv=sb/root ..."
append="coreos.oem.id=qemu ..."

If the user did use rpm-ostree kargs --{append,delete,replace} or ostree admin --karg*, then the merge deployment would become the source of truth.

[rfairley]

Thinking about rpm-ostree integration (Re: https://github.com/ostreedev/ostree/issues/479#issuecomment-472920616):

For rpm-ostree treefiles, I'm thinking we could add a field e.g. default-kargs or just kargs, where the user specifies a list of key-value pairs for each karg (arg, value). The key-value pairs would be written to /usr/lib/ostree/kargs at compose time, and that file would be committed to the ostree.

It looks like an easier way to add /usr/lib/ostree-boot/kargs at compose time is to use the add-files field (https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/treefile.md#treefile). Will test this out.

[jlebon]

If the user did use rpm-ostree kargs --{append,delete,replace} or ostree admin --karg*, then the merge deployment would become the source of truth.

Hmm, I think we could at least optimize rpm-ostree kargs --append=newkey to write to /etc/ostree/kargs so we stay managed by the baked configs?

Should the host config (/etc/ostree/kargs) have an equivalent of rpm-ostree kargs --delete when merging/overriding with the base config

Yeah, that's a good question. Clearly, adding new kargs is the common case, though support for "subtractive" semantics would definitely be nice. One alternative to your suggestion would be to support /usr/lib/ostree-boot/kargs.d instead of just kargs where each file in there is for a separate key (and the file could have multiple new-line separated values for keys like console). Then "removing" a karg just means dropping an empty file in /etc/ostree/kargs.d of the same name. It's a bit more complex, though the whole conf.d in /etc overrides conf.d in /usr paradigm is pretty common these days.

That said, I think there can be a path from the current approach you have in #1836 to something like that without breaking existing semantics.

[rfairley]

I think we could at least optimize rpm-ostree kargs --append=newkey to write to /etc/ostree/kargs so we stay managed by the baked configs?

Right, I hadn't thought about the append case v.s. replace/delete. rpm-ostree kargs --append= would fit nicely writing to /etc/ostree/kargs.

I like the /usr/lib/ostree-boot/kargs.d idea. Users could put multiple kargs in their own files too (say /etc/ostree/kargs.d/custom-kargs), so the kargs could be managed as a group if they chose. Shipping kargs in /usr/lib/ostree-boot/kargs.d as one-karg-per-file snippets would allow users to turn on/off each one while still having the other kargs managed by updates to the repo. The replace semantic would be satisfied too by dropping the empty file and creating another file with the same karg in /etc/ostree/kargs.d.

If we wanted to unify the rpm-ostree kargs commands further and avoid the need for something like https://github.com/ostreedev/ostree/pull/1836/commits/65fe4e952dd87688059689c59c18eebb5e694c30, we could map --replace and --delete as well as --append to the respective operations in/etc/ostree/kargs.d and always generate from there? To do this would rely on specific naming for each karg snippet, e.g. /etc/ostree/kargs.d/net.ifnames for the net.ifnames karg (which I guess is the obvious choice for a single karg file).

[rfairley]

Following up:

It looks like an easier way to add /usr/lib/ostree-boot/kargs at compose time is to use the add-files field (https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/treefile.md#treefile). Will test this out.

I tested adding the kargs through add-files in rpm-ostree treefile (adding

add-files:
  - - kargs
    - /usr/lib/ostree-boot/kargs

to https://github.com/coreos/fedora-coreos-config/blob/6ca5469cbfd871536f71903d6307b6bf95a40687/fedora-coreos-base.yaml in FCOS).

The /usr/lib/ostree-boot/kargs file is created fine (i.e. add-files works as expected), but making another deployment (e.g. ostree admin deploy) is necessary to get the kargs from that file into the BLS config. Will look to see what's needed to have the kargs reflected in the BLS right after the compose.

[cgwalters]

If you want to test this with cosa or Anaconda in general, the heavyweight method would be to use lorax to rebuild the anaconda ISO. However what I'd do for faster iteration is a hack like:

Get a tarball of the binaries in git: make install DESTDIR=install/ && tar -C install -czf ostreebuild.tar.gz

And then mv ostreebuild.tar.gz /path/to/cosa-builddir/repo (move it into the ostree repo in your cosa builddir) and use something like this in cosa:

diff --git a/src/virt-install b/src/virt-install
index c829fca0..adc9cfe9 100755
--- a/src/virt-install
+++ b/src/virt-install
@@ -144,6 +144,9 @@ if args.ostree_repo:
     assert args.ostree_ref
     ks_tmp.write(f"""
 ostreesetup --nogpg --osname={args.ostree_stateroot} --remote={args.ostree_remote} --url=http://{ipv4}:{PORT}/ --ref="{args.ostree_ref}"
+%pre
+curl -L http://{ipv4}:{PORT}/ostreebuild.tar.gz | (cd / && tar xf -)
+%end
     """)

 if args.fs9p:

[rfairley]

Ah cool! Thanks, giving this a try.

[cgwalters]

Ultimately though we want the remove anaconda work as that would mean we use the ostree binaries from the container just like everything else.

Also one trap you may hit is that cosa switched to f30 but the anaconda build is still f29...it might work to overlay f30-built binaries on anaconda's f29 environment but if you run into e.g. glibc/openssl issues you might need to build ostree in a f29 container.

[rfairley]

The curl command worked - just made a slight change (https://github.com/rfairley/coreos-assembler/commit/4f157f38d68204eb48acff0c233982d59628b1b3) (the request handler failed with the initial edit, with Connection reset by peer before I think from the pipe somehow failing? not sure on this, so just separated the commands quickly).

However - the kernel args still don't show in the BLS config immediately after cosa build && cosa run. Now looking more closely at the anaconda code it might be that ostree admin instutil set-kargs is overwriting the kargs set by ostree admin deploy. I initially left this in #1836 as the --help output for instutil showed Deprecated - but should probably handle this for systems that still use anaconda.

I think ostree admin instutil set-kargs could be treated like rpm-ostree kargs and change the source of truth if instutil is used by an installer. Or, with the kargs.d way, write to /etc/ostree/kargs.d.

Right now anaconda rewrites the kargs completely using instutil - so any defaults in /usr/lib/ostree-boot/kargs would need to be "deleted" if using kargs.d. I think this could be done by writing an empty file for any default kargs existing in /usr/lib/ostree-boot/kargs.d except for the ones specified by the installer in ostree admin instutil set-kargs.

[rfairley]

Planning to re-iterate on this - just want to flesh out an implementation based on the kargs.d idea:

Generating BLS config from kargs.d directories

The following directories are added:

/etc/ostree/kargs.d/ # host config
/usr/lib/ostree-boot/kargs.d/ # base config

All files in the above directories are appended when generating a new BLS config. A file of name <key> in the host config will override a file of name <key> in the base config.

Files existing in those directories must have the name <key> and contain a string of the form <key>=<value1>,<value2>,..., where <key> is the name of the kernel argument. For example:

# /usr/lib/ostree-boot/kargs.d/console
console=ttyS0,115200n8

A file like the following will give the karg <key> empty arguments:

# /usr/lib/ostree-boot/kargs.d/console
console=

An empty file of name <key> in the host config will effectively delete the karg <key> from the base config, as the file /usr/lib/ostree-boot/kargs.d/<key> will not be read.

Supporting existing semantics

rpm-ostree kargs --replace=KEY=VALUE=NEWVALUE: copy /usr/lib/ostree-boot/kargs.d/KEY to /etc/ostree/kargs.d/KEY, and perform the replace operation on the string in that file. If the file /etc/ostree/kargs.d/KEY exists already, do not copy from /usr/lib/ostree-boot/kargs.d/KEY. Regenerate the BLS config.

rpm-ostree kargs --append=KEY=VALUE: same as above, but perform the append operation instead of replace.

rpm-ostree kargs --delete=KEY=VALUE: write an empty file to /etc/ostree/kargs.d/KEY, and regenerate the BLS config.

If --deploy-index or --import-proc-cmdline are used, then import those kargs into the host config, perform the modifications the user specifies on the host config, then regenerate the BLS config.

If --editor is specified, then import the edited result into the host config, and regenerate the BLS config.

An import looks like this:

1. Delete base config kargs by dropping an empty file in /etc/ostree/kargs.d/ for every file in /usr/lib/ostree-boot/kargs.d/*.
2. Populate /etc/ostree/kargs.d/ by reading the imported kargs and writing files containing the individual kargs.

Similar options in the ostree command line ostree admin deploy --{karg=NAME=VALUE,karg-append=NAME=VALUE,karg-prog-cmdline} behave like the rpm-ostree options described above. ostree admin deploy --karg-none will delete all the base config kargs by writing empty files to /etc/ostree/kargs.d/.

ostree admin instutil set-kargs will write its result to the host config /etc/ostree/kargs.d/ before regenerating the BLS config.

Upgrade flag

By default, new composes will write a flag like ostree-kargs-generated-from-config=true to the BLS config. The presence of this flag indicates to use the kargs.d directories. If the flag is not present, then copy the kargs from the previous deployment. Existing users can add this flag once they are ready to use the committed kargs. (Brought up in this comment). The flag in this commit is wrong - will rework this.

In summary, command behavior is the same, but with a layer of first writing kargs information into files in /etc/ostree/kargs.d/ and regenerating from /etc/ostree/kargs.d/ and /usr/lib/ostree-boot/kargs.d/. This should allow users to choose which kargs use the defaults from /usr/lib/ostree-boot/kargs.d/ and which are custom-configured, on a per karg basis.

[rfairley]

After recent discussion on kargs - it seems like the mechanism here will need to support ordering in some way.

One way is that each file in /etc/ostree/kargs.d is treated more like a general snippet with a number prefixed so the snippets are read in alphanumeric order by ostree. The motivating example for this is the alternating hugepagesz=1M hugepages=8 hugepagesz=2G hugepages=2 setting (linked in the discussion above). To ship this in karg snippets, one would include files like 0001_hugepagesz, 0002_hugepages, 0003_hugepagesz, 0004_hugepages (or combine each hugepagesz/hugepages pair in one snippet).

Files existing in those directories must have the name and contain a string of the form =,,..., where is the name of the kernel argument.

This restriction shouldn't be needed. E.g., an arbitrary name like /etc/ostree/kargs.d/0001_localhostkargs containing different karg keys e.g. console=tty0 nosmt should be permissible. This would give the users more choice to group their custom kargs in one file if they choose - which is helpful for kargs with different keys that are dependent on each other.

When FCOS ships kargs, FCOS should still ship 1 karg per file (or possibly group dependent ones) in /usr/lib/ostree-boot/kargs.d so that users can choose which kargs to disable individually by writing an empty file / null symlink in /etc/ostree/kargs.d. For FCOS today, this would look something like the following:

# /usr/lib/ostree-boot/kargs.d
4000_console
4001_console
4002_rootflags
4003_rw
4004_mitigg
4005_ignition.platform.id

Having more general snippets does make optimizing the rpm-ostree kargs command line to stay managed by the snippets a bit trickier. E.g., for rpm-ostree kargs --replace KEY=VALUE=NEWVALUE, ostree would need to essentially grep the kargs.d directories to find which file KEY=VALUE exists in, and edit that file (rather than having certainty that the karg exists in /etc/ostree/kargs.d/KEY). However, searching for the matching file shouldn't be too expensive of an operation (O(kn) with n being number of kargs snippets, and k the size of the largest snippet, which should be small). rpm-ostree kargs --delete KEY=VALUE would need to do a similar search. rpm-ostree kargs --append could just write a new snippet, incrementing the ordering prefix.

[rfairley]

In #1836 I was initially planning on optimizing ostree admin deploy --karg=, --karg-append=, etc. (which COSA uses since removing Anaconda https://github.com/coreos/coreos-assembler/pull/556/files#diff-2d426a046eb99541d19814f8fac698bbR54) to write snippets to /etc/ostree/kargs.d in order to modify kargs. However in the case of checking existence of this directory before doing a reboot (https://github.com/coreos/ignition-dracut/pull/86), this would trigger an unnecessary reboot even if Ignition did not write snippets there.

(I didn't realize this when suggesting it at https://github.com/coreos/ignition-dracut/issues/81#issuecomment-494888494).

One way to get around this could be to have COSA bake the kargs snippets in a new commit, i.e. under /usr/lib/ostree-boot/kargs.d (or use add-files in the FCOS treefile) to avoid the /etc/ostree/kargs.d path. However this is restrictive - there should really be a way to specify snippets in FCOS at install time without needing a new commit (i.e. write in /etc/ostree/kargs.d), and without needing to specify kargs snippets through an Ignition config.

Ignition could write a magic file e.g. /etc/ostree/kargs.d/0000_ignition-reboot - however we want to avoid special handling in Ignition. Users also shouldn't be required to write to a special directory in their Ignition config e.g. /etc/ostree/kargs.d/ignition/<user's kargs snippets here> for the needed reboot to happen (if the reboot was triggered on /etc/ostree/kargs.d/ignition instead).

[cgwalters]

Was thinking about this more since doing factory reset support.

One model might be to suggest people to add default kernel arguments into the kernel binary; this is already supported with CONFIG_CMDLINE. For distributions like Fedora that want to build a generic kernel image that might be used multiple ways (e.g. FCOS vs other systems), it probably wouldn't be hard to patch the kernel to reserve some fixed space for arguments that one could write into the binary. Or, maybe instead we support "re-linking" the binary with some default args?

This would break Secure Boot signatures, but on the other hand, not validating kernel arguments is often an easy way to execute arbitrary code too....

[cgwalters]

When FCOS ships kargs, FCOS should still ship 1 karg per file (or possibly group dependent ones) in /usr/lib/ostree-boot/kargs.d

The current model is now /usr/lib/modules/<kver> - if we defined a file format there, it could actually also apply to traditional systems.

[nullr0ute]

This would break Secure Boot signatures, but on the other hand, not validating kernel arguments is often an easy way to execute arbitrary code too....

You can deal with validation of kernel args to ensure they don't change with IMA/tpm2.