do not use tmps for /tmp and /run

[dxlr8r]

k3OS is made to run systems with little memory, and on such devices memory management and preservations is very important.

Currently /tmp & /run are mounted in memory, allocating 10% of total system memory to the filesystem.

Content stored to this filesystem not only consumes that amount of memory, but it also makes memory management difficult, as OOM-killer has no domain here.

I suggest storing these mountpoints to disk. This can be done by:

• altering https://github.com/rancher/k3os/blob/4738bb8f7c4bfcb22656e536077d71bd30c1138f/pkg/cli/rc/rc.go#L180-L182

Or to use /etc/fstab, and remove the functionality from rc.go all together. I would prefer the latter.

[dweomer]

@dxlr8r this is not an unreasonable suggestion. Can you elaborate on specific use-cases that are difficult (or impossible) to implement with things as they currently are? I do suspect that a non-trivial number of existing use-cases might be incur unexpected breakage with ephemeral state under /run persisting through reboots. I agree that would be nice if k3OS was less opinionated here and/or more configurable. However, I do think it might be possible to unmount these locations early via init_cmd entries in your config.yaml, i.e.:

# untested, use at your own risk
init_cmd:
- mountpoint -q /tmp && umount /tmp
- mountpoint -q /run && umount /run

It is possible that these as init_cmd operations come too early but I worry that as boot_cmd they would come too late. Worth a try!

[dxlr8r]

@dxlr8r this is not an unreasonable suggestion. Can you elaborate on specific use-cases that are difficult (or impossible) to implement with things as they currently are?

This makes it more difficult to do capasity management. For example a large write to /tmp might lead to containers begin evicted.

I do suspect that a non-trivial number of existing use-cases might be incur unexpected breakage with ephemeral state under /run persisting through reboots.

That can be fixed by recreating/cleaning the mountpoint each reboot.

I agree that would be nice if k3OS was less opinionated here and/or more configurable. However, I do think it might be possible to unmount these locations early via init_cmd entries in your config.yaml, i.e.:

# untested, use at your own risk
init_cmd:
- mountpoint -q /tmp && umount /tmp
- mountpoint -q /run && umount /run

It is possible that these as init_cmd operations come too early but I worry that as boot_cmd they would come too late. Worth a try!

That might be possible, and I might get time to look into that. But in general, I feel sane defaults should be a part of the core, and not being added to config.yaml.