Document and adjust `Hoster` to be executed in a Jail

yaroslav-gwit commented 1 year ago

Running Hoster in a Jail will bring a "true multi-tenancy" to life. You'll be able to simply create a new Jail, install Hoster in it, start the SSH service and pass the credentials down to the end user. Even if there are multiple users on the same host, they would not be able to know that someone else is using the same node and running even more VMs on it.

For this feature to be implemented, I'll have to create some static files in the Jail's FS to indicate resource constrains applied by rctl and link those with Hoster. Otherwise the end user might over-provision and crash their Jail.

I call this feature Hoster-As-A-Service 🔥 🤣🤣

Future refs: https://github.com/churchers/vm-bhyve/issues/267 https://github.com/lattera/articles/blob/master/freebsd/2018-10-27_jailed_bhyve/article.md https://forums.freebsd.org/threads/bhyve-inside-jail.79935/ https://news.ycombinator.com/item?id=26742588

yaroslav-gwit commented 4 months ago

Here is the rough implementation plan.

Install process:

Create a Hoster-in-a-Jail installation script (or an Ansible playbook), because the normal node-init.sh would not work

Related to `jail_config.json`

ZFS Mounts:

Add a new jail configuration flag, that will allow mounting Hoster datasets within a Jail automatically

"mount_zfs_datasets": ["tank/vm-encrypted/hoster-jail/pool", "tank/client_pool_23452210878"]

It will execute these 2 commands on each Jail start:

zfs set jailed=on tank/vm-encrypted/hoster-jail/pool
zfs jail netbirdCheck tank/vm-encrypted/hoster-jail/pool

Add a new configuration flag `jailed_hoster`

This flag will make sure hoster binary gets the information normally not available within the Jail (e.g. CPU Info, etc) from a new location. It will also give me the ability to add some if statements when a VM or a Jail starts within the Jail, to apply a Jail-special set of parameters, etc.

Here is how it'll look like in the config file:

"jailed_hoster": true

Apply new Host `sysctl`s

sysctl security.jail.vmm_allowed=1
sysctl security.jail.mount_zfs_allowed=1
sysctl security.jail.mount_devfs_allowed=1
sysctl security.jail.mount_tmpfs_allowed=1
sysctl security.jail.mount_procfs_allowed=1
sysctl security.jail.mount_zfs_allowed=1
sysctl -w security.bsd.unprivileged_proc_debug=1

Start `devfs` service on the Host

service devfs enable
service devfs start

Add the appropriate devfs permissions to /etc/devfs.rules

# Allow Hoster-in-a-Jail to use PF
[devfsrules_jail_unhide_pf=10]
add path 'pf*' unhide

TBD:

Check why internal Jails don't start
Check if self running in Jail, and apply a set of modified hoster init options
Set the DNS server to 127.0.0.1 at the Jail boot time
Generate the public network interface for the Jailed PF dynamically
Make sure all these options are available in the jail_custom_parameters.conf:
```
zfs;
zfs.mount_snapshot;
```

allow.vmm; allow.set_hostname = 0; allow.raw_sockets; allow.sysvipc; allow.socket_af; allow.quotas; allow.raw_sockets;

allow.mount; allow.mount.devfs; allow.mount.tmpfs; allow.mount.procfs; allow.mount.zfs;

devfs_ruleset = 10; # our PF ruleset enforce_statfs = 1;

mount.devfs; mount.fdescfs; mount.procfs;

yaroslav-gwit commented 4 months ago

This was a really nice idea initially, but I've hit way too many blockers along the way (at least for now?). Rctl doesn't work within a Jail, a bunch of hardware based checks related to monitoring too, and it will require too much effort to support all these exceptions/workarounds.

Closing for now. May come back to it later at some point, when there is more free time on my hands.

yaroslav-gwit / HosterCore