cloud-init via cdrom provider - working example?

[docbobo]

Version (k3OS / kernel) k3os version v0.21.1-k3s1r0

Architecture x86_64

Describe the bug Maybe it's a bug, maybe it's not. I am totally unclear on how exactly the cloud-init cdrom provider is supposed to be working. Here's my assumption:

• Install k3os to disk with some settings and use the result as some kind of template.
• Start a machine based on that template with a cloud-init image containing customizations
• Have k3os pick up those settings

Actually, I don't see why the cdrom datasource were to be present if things are working that way, but maybe I am missing something obvious.

So when I was trying this for the first time, I was running into the same issues reported in rancher/k3os#619, which lead me to https://github.com/linuxkit/linuxkit/blob/45429986290f3811b6da093e8fdbbaffce96616e/pkg/metadata/main.go#L208. Apparently, this line of code is expecting a JSON file, not a YAML, which explains the reported parsing error. So I was guessing, that some conversion from YAML to JSON needs to happen. Unfortunately, I didn't find much information about that.

Desperately trying to get this to work, I've started using the JSON documented at https://github.com/linuxkit/linuxkit/blob/master/docs/metadata.md as an example. And it does exactly what's described there - it creates a directory structure under /run/config containing JSON files with the corresponding content.

So how exactly can this be used to update things like

• ssh keys
• k3s settings
• etc.

Maybe an actual working example could clear up things

To Reproduce n/a

Expected behavior n/a

Actual behavior n/a

Additional context n/a

[docbobo]

Okay, I guess I found the answer. According to https://github.com/rancher/k3os/blob/4738bb8f7c4bfcb22656e536077d71bd30c1138f/pkg/config/read_cc.go, content from the following locations will be picked up:

• /run/config/local_hostname
• /run/config/ssh/authorized_keys
• /run/config/userdata

So with a user-data file structure like this:

{ "local_hostname": { "Perm": "0600", "Content": "hostname" }, "ssh": { "Entries": { "authorized_keys": { "Perm": "0600", "Content": "ssh-rsa ssh-ed25519 ..." } } }, "userdata": { "Perm": "0600", "Content": "#cloud-init\n" } }

I was at least able to update ssh keys and hostname. Not sure if userdata will actually work as expected...

That said, I probably don't have to do all of this manually. So what part of the documentation have I been missing out on? ;)

[docbobo]

Or is this something that a regular cloud-init tool will do for me? Because I had to create the image manually via mkisosfs, due to some linuxkit incompatibility with -rock...

[dweomer]

So, what is going on here is that k3OS is leveraging the linuxkit metadata cdrom datasource to scan any /dev/sr* devices with userdata or config at the root of the disk and use the content found there as the userdata to be processed (copied to /run/config/userdata) by multiple invocations of k3os config during startup:

• https://github.com/rancher/k3os/blob/v0.21.1-k3s1r0/overlay/libexec/k3os/bootstrap#L44
• https://github.com/rancher/k3os/blob/v0.21.1-k3s1r0/overlay/libexec/k3os/boot#L72
• https://github.com/rancher/k3os/blob/v0.21.1-k3s1r0/overlay/etc/init.d/ccapply

As specified here, https://github.com/rancher/k3os#configuration, k3os config processes yaml config files in the following order:

/k3os/system/config.yaml
/var/lib/rancher/k3os/config.yaml
/var/lib/rancher/k3os/config.d/*

followed by /run/config/userdata which you can see here: https://github.com/rancher/k3os/blob/v0.21.1-k3s1r0/pkg/config/read.go#L41-L45. The last config in wins with maps getting merged but all other values being replaced (including arrays/slices! this can lead to some unexpected results).

---

For a working example, copy a sample config.yaml to user-data into a directory to be used as the root of an isofs then do the mkisofs needful. Optionally, a colleague wrote https://github.com/luthermonson/quiso to make this super easy.

[docbobo]

Thanks, @dweomer. I think with the information you just provided and the things I found out myself, I do have a pretty good understanding how it's working now. As a fact of the matter, I have this working as expected. But...

The final thought you were giving at the end of the your is exactly what I (and probably everybody else) thought how things should be working. Unfortunately, they are not. So let me provide some more information.

quiso

I tried as you've suggested and created the image with quiso. Both meta-data and user-data were kept quite simple. Here's the result:

k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
No metadata/userdata found. Bye
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /home/rancher/mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data  user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
#cloud-init
#quiso

Apparently, the quiso created image is not even recognized by the metadata process. I don't know why that's the case, but the image itself is notably smaller (~46k) than the ones created with mkisofs (~362k)

mkisofs

My next attempt was using the exact same files with mkisofs. Here's the result:

k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
CDROM /dev/sr0: Probe succeeded
Could not unmarshall userdata: invalid character '#' looking for beginning of value
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data  user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
#cloud-init

As you can see, metadata finds the user-data file this time, but fails to parse it (see also #619). As I've written above, this is because metadata expects a JSON file, not a YAML one.

mkisofs with a user-data JSON

The last snippet shows how I managed to create an image that is recognized properly. It requires the conversion of the user-data file into a JSON that actually results in writing a userdata file in /run/config:

k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
CDROM /dev/sr0: Probe succeeded
k3os-ca3a63fdb041 [/home/rancher]$ cat /run/config/userdata
#cloud-init
#JSON
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data  user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
{
  "userdata": {
    "Perm": "0600",
    "Content": "#cloud-init\n#JSON\n"
  }
}

Conclusion

I am sorry, but copying a sample config.yaml to user-data and creating an ISO image out of it does not cut it 😉

---

I've created a small shell script that picks up hostname, authorized_keys, and config.yaml files to dump out the appropriate user-data content, if anybody's interested:

#!/bin/bash

hostname=$(<hostname)
userdata=$(<config.yaml)
ssh_authorized_keys=$(<authorized_keys)
jq -n --arg hostname "$hostname" --arg ssh_authorized_keys "$ssh_authorized_keys" --arg value0 "$userdata" '{"local_hostname":{"Perm":"0600","Content":$hostname}, "ssh":{"Entries":{"authorized_keys":{"Perm":"0600","Content":$ssh_authorized_keys}}}, "userdata":{"Perm":"0600","Content":$value0}}'

[dweomer]

As you can see, metadata finds the user-data file this time, but fails to parse it (see also #619). As I've written above, this is because metadata expects a JSON file, not a YAML one.

Yes, linuxkit metadata will fail to parse YAML user-data but it will copy the contents to /run/config/userdata regardless. This is the intended/expected behavior.

[docbobo]

D'uh. I had that suspicion when I ran the tests above but apparently mix something up, when I tried to confirm it. You are right, it does - and so does the source code say.

Now, I have to admit that the intended/expected behavior is hugely confusing...

At least now we have on post that describes all that - plus a mechanism to build a cloud-init image that does not produce that error 😉

Any idea, why the quiso image did not work?

[docbobo]

One more thing: is there any (additional) advantage to the JSON file? I noticed that it updates the hostname and ssh keys in a somewhat different way...

[dweomer]

One more thing: is there any (additional) advantage to the JSON file? I noticed that it updates the hostname and ssh keys in a somewhat different way...

Not really, it is up to you which you find more advantageous.