Closed docbobo closed 2 years ago
Okay, I guess I found the answer. According to https://github.com/rancher/k3os/blob/4738bb8f7c4bfcb22656e536077d71bd30c1138f/pkg/config/read_cc.go, content from the following locations will be picked up:
So with a user-data file structure like this:
{ "local_hostname": { "Perm": "0600", "Content": "hostname" }, "ssh": { "Entries": { "authorized_keys": { "Perm": "0600", "Content": "ssh-rsa ssh-ed25519 ..." } } }, "userdata": { "Perm": "0600", "Content": "#cloud-init\n" } }
I was at least able to update ssh keys and hostname. Not sure if userdata will actually work as expected...
That said, I probably don't have to do all of this manually. So what part of the documentation have I been missing out on? ;)
Or is this something that a regular cloud-init tool will do for me? Because I had to create the image manually via mkisosfs, due to some linuxkit incompatibility with -rock
...
So, what is going on here is that k3OS is leveraging the linuxkit metadata cdrom
datasource to scan any /dev/sr*
devices with userdata
or config
at the root of the disk and use the content found there as the userdata to be processed (copied to /run/config/userdata
) by multiple invocations of k3os config
during startup:
As specified here, https://github.com/rancher/k3os#configuration, k3os config
processes yaml config files in the following order:
/k3os/system/config.yaml
/var/lib/rancher/k3os/config.yaml
/var/lib/rancher/k3os/config.d/*
followed by /run/config/userdata
which you can see here: https://github.com/rancher/k3os/blob/v0.21.1-k3s1r0/pkg/config/read.go#L41-L45. The last config in wins with maps getting merged but all other values being replaced (including arrays/slices! this can lead to some unexpected results).
For a working example, copy a sample config.yaml
to user-data
into a directory to be used as the root of an isofs then do the mkisofs
needful. Optionally, a colleague wrote https://github.com/luthermonson/quiso to make this super easy.
Thanks, @dweomer. I think with the information you just provided and the things I found out myself, I do have a pretty good understanding how it's working now. As a fact of the matter, I have this working as expected. But...
The final thought you were giving at the end of the your is exactly what I (and probably everybody else) thought how things should be working. Unfortunately, they are not. So let me provide some more information.
I tried as you've suggested and created the image with quiso. Both meta-data
and user-data
were kept quite simple. Here's the result:
k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
No metadata/userdata found. Bye
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /home/rancher/mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
#cloud-init
#quiso
Apparently, the quiso created image is not even recognized by the metadata
process. I don't know why that's the case, but the image itself is notably smaller (~46k) than the ones created with mkisofs
(~362k)
My next attempt was using the exact same files with mkisofs
. Here's the result:
k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
CDROM /dev/sr0: Probe succeeded
Could not unmarshall userdata: invalid character '#' looking for beginning of value
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
#cloud-init
As you can see, metadata
finds the user-data
file this time, but fails to parse it (see also #619). As I've written above, this is because metadata
expects a JSON file, not a YAML one.
The last snippet shows how I managed to create an image that is recognized properly. It requires the conversion of the user-data
file into a JSON that actually results in writing a userdata
file in /run/config
:
k3os-ca3a63fdb041 [/home/rancher]$ metadata
Trying CDROM /dev/sr0
CDROM /dev/sr0: Probe succeeded
k3os-ca3a63fdb041 [/home/rancher]$ cat /run/config/userdata
#cloud-init
#JSON
k3os-ca3a63fdb041 [/home/rancher]$ mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
k3os-ca3a63fdb041 [/home/rancher]$ ls /mnt
meta-data user-data
k3os-ca3a63fdb041 [/home/rancher]$ cat /mnt/user-data
{
"userdata": {
"Perm": "0600",
"Content": "#cloud-init\n#JSON\n"
}
}
I am sorry, but copying a sample config.yaml
to user-data
and creating an ISO image out of it does not cut it 😉
I've created a small shell script that picks up hostname
, authorized_keys
, and config.yaml
files to dump out the appropriate user-data
content, if anybody's interested:
#!/bin/bash
hostname=$(<hostname)
userdata=$(<config.yaml)
ssh_authorized_keys=$(<authorized_keys)
jq -n --arg hostname "$hostname" --arg ssh_authorized_keys "$ssh_authorized_keys" --arg value0 "$userdata" '{"local_hostname":{"Perm":"0600","Content":$hostname}, "ssh":{"Entries":{"authorized_keys":{"Perm":"0600","Content":$ssh_authorized_keys}}}, "userdata":{"Perm":"0600","Content":$value0}}'
As you can see,
metadata
finds theuser-data
file this time, but fails to parse it (see also #619). As I've written above, this is becausemetadata
expects a JSON file, not a YAML one.
Yes, linuxkit metadata
will fail to parse YAML user-data but it will copy the contents to /run/config/userdata
regardless. This is the intended/expected behavior.
D'uh. I had that suspicion when I ran the tests above but apparently mix something up, when I tried to confirm it. You are right, it does - and so does the source code say.
Now, I have to admit that the intended/expected behavior is hugely confusing...
At least now we have on post that describes all that - plus a mechanism to build a cloud-init
image that does not produce that error 😉
Any idea, why the quiso image did not work?
One more thing: is there any (additional) advantage to the JSON file? I noticed that it updates the hostname and ssh keys in a somewhat different way...
One more thing: is there any (additional) advantage to the JSON file? I noticed that it updates the hostname and ssh keys in a somewhat different way...
Not really, it is up to you which you find more advantageous.
Version (k3OS / kernel) k3os version v0.21.1-k3s1r0
Architecture x86_64
Describe the bug Maybe it's a bug, maybe it's not. I am totally unclear on how exactly the cloud-init cdrom provider is supposed to be working. Here's my assumption:
Actually, I don't see why the cdrom datasource were to be present if things are working that way, but maybe I am missing something obvious.
So when I was trying this for the first time, I was running into the same issues reported in rancher/k3os#619, which lead me to https://github.com/linuxkit/linuxkit/blob/45429986290f3811b6da093e8fdbbaffce96616e/pkg/metadata/main.go#L208. Apparently, this line of code is expecting a JSON file, not a YAML, which explains the reported parsing error. So I was guessing, that some conversion from YAML to JSON needs to happen. Unfortunately, I didn't find much information about that.
Desperately trying to get this to work, I've started using the JSON documented at https://github.com/linuxkit/linuxkit/blob/master/docs/metadata.md as an example. And it does exactly what's described there - it creates a directory structure under /run/config containing JSON files with the corresponding content.
So how exactly can this be used to update things like
Maybe an actual working example could clear up things
To Reproduce n/a
Expected behavior n/a
Actual behavior n/a
Additional context n/a