(pronounced nibble)
Nyble solves the problem of configuration drift and bootable image divergence across fleets of physical/virtual machines, by creating immutable OS images. These images can be network (PXE, SAN), USB, and local disk booted. Nyble instances, the booted version of a Nyble image, may be configured programmatically upon boot, using any mechanism you prefer for configuration management.
With a Nyble based OS image, you cannot get image divergence or configuration drift across your fleet. It solves this by creating a fixed bootable initramfs artifact for a particular linux distribution. The initramfs creation is driven by the Makefile in this repo.
Far more importantly, you should never have an errant driver or configuration stop a node from correctly booting. You should always be able to come up to an operational state of an OS, and apply configuration
Nyble sits atop the basic distro packaging mechanisms (apt, yum) and adds
specific configuration, tools, drivers, and packages that you may modify.
One should be able to build an image from any distro as long as the
relevant target distro's tools are installed.
Nyble will build an image from the tools into a temporary scratch space. This build will then be snapshot into a compressed tar image using tar and a compressor. After that the image will be incorporated into an initramfs, after adjusting the initramfs configuration to include the tarball, and the elements required to unpack it.
You can select different compressors by adding a COMP=pigz
or similar on the
build line. Your options are gzip, bzip2, lbzip2, pigz, xz, pxz, zstd. Current
default is set to zstd, given how fast it compresses and decompresses compared
to the others. Compression ratios are similar across all of the compressors.
This effectively limits the size of the tarball to 2GB, as cpio cannot handle larger files at this moment.
Several important variables used for the build are
DISTRO : which distribution you will use as the base of your image. Current choices are debian12, debian11, ubuntu18.04, ubuntu20.04, ubuntu22.04, Rocky9. Rocky8 is in process.
TARGET : top level scratch directory for building the image. Defaults to /mnt/root
.
NYBLE_KERNEL : 0 or 1, with 0 indicating that the build should use the distro provided default kernel. This does mean you can build these images to be almost entirely based upon the distro itself, with minor modifications to some of the startup scripts. These modifications are necessary to run as a ramdisk booted OS. The system will not likely function without them.
The Makefile includes distro specific configuration in OS/$DISTRO/{base,config}.conf
. The Makefile uses 1 target, finalizebase
which should be the last target in OS/$DISTRO/config.conf
. The OS/$DISTRO/base.conf
portion of
the distro specific included Makefile should handle all of the base distro
package installation, and kernel installation. The OS/$DISTRO/config.conf
should handle all of the post installation configuration, additional driver,
package, and feature installation.
In short
# # create a debian12 based distro
make DISTRO=debian12
# # create a debian11 based distro
make DISTRO=debian11
# # create a ubuntu20.04 based distro
make DISTRO=ubuntu20.04
# # create a ubuntu22.04 based distro
make DISTRO=ubuntu22.04
# # create a Rocky9 based distro
make DISTRO=rocky9
# # install to a physical disk/raid/device you have mounted at /mnt/root
make PHYSICAL=1 [DISTRO=...]
# note this isn't working right now
This will take some time even with a good internet connection and a close mirror. It will, upon completion, leave two specific files you need in /mnt/root/boot. These will be named initramfs-ramboot-$SOME_KERNEL_VERSION and vmlinuz-$SOME_KERNEL_VERSION. You should preserve these as the build artifacts.
You may take these two files, and using virt-manager (or qemu) do a direct kernel boot. Add in kernel boot arguments like this:
root=ram rootfstype=ramdisk simplenet=1 verbose udev.children-max=4 rootpw=nyble
You will have a nice console to a ramdisk based full distro running in about 10GB of RAM. You can tweak that amount by adding a 'ramdisksize=X ' option for values of X from 6-32. Units are GB.
These files may be served by PXE (I use ipxe for this), or placed into a systems /boot as a rescue distribution. For that, I create an /etc/grub.d/40_custom file with these contents
#!/bin/sh
cat <<EOF
menuentry 'Nlytiq Nyble Ubuntu 18.04 hwe' {
linux /boot/nyble/vmlinuz-5.3.0-40-generic root=ram rootfstype=ramdisk simplenet=1 verbose udev.children-max=4 rootpw=nyble ramdisktype=zram ramdisksize=12
initrd /boot/nyble/initramfs-ramboot-5.3.0-40-generic
}
EOF
Of course, modify your kernel version in the initrd and kernel lines to suit what you have built.
You can execute up to a specific target, for example fb_final
by running
make fb_final
. You can continue the image build process by running make
.
You can inspect any variable in the Makefile, by using make print-VARIABLE_NAME
# make print-DISTRO
DISTRO = debian12
# make DISTRO=debian11 print-DISTRO
DISTRO = debian11
# make print-TARGET
TARGET = /mnt/root
# make TARGET=/outerspace print-TARGET
TARGET = /outerspace
# make print-NYBLE_KERNEL
NYBLE_KERNEL = 0
# make NYBLE_KERNEL=1 print-NYBLE_KERNEL
NYBLE_KERNEL = 1
Upon successful completion of the build, you will have a kernel and initramfs located in ${TARGET}/boot/
that you may use for PXE booting.
Since git does not allow for large BLOB artifacts, you need to store them in a different location which can be easily pulled during build. This is
The Makefile includes distro specific configuration in OS/$DISTRO. There are two specific mechanisms for adding functionality, using the drivers or packages directory off of the main repository directory.
You will need a machine with a fast network, and at least 64 GB RAM. This builds the image in a ramdisk, which you can copy out to permanent storage. You can alter this behavior, by changing the TARGET= variable in the Makefile.
git clone https://github.com/joelandman/nyble
cd nyble
# edit the kernel/kernel.conf ,
# urls.conf ,
# and OS/${DISTRO}/distro_urls.conf as needed to
# point to local repos and kernel repos
# Edit config/all.conf to adjust default features/functionality
make [FEATURE_1=0|1] [FEATURE_2=0|1] ... [FEATURE_N=0|1]
Support for a bootable usb exists if you use the usb
target. That is, to
make a bootable USB drive image
```make usb [DISTRO=...]```
at the end of the process, you will find a file named nyble.usb
in your directory.
To use this, copy it to a USB drive. The easiest way to do this is using the
dd command in linux.
```umount /dev/$USB ; dd if=nyble.usb of=/dev/$USB oflag=direct bs=1M status=progress```
You may determine your USB drive device using the lsscsi
command,
```
joe@gandalf:~$ lsscsi
[0:0:0:0] disk ATA WDC WDS250G2B0B- 30WD /dev/sda
[2:0:0:0] disk ATA Samsung SSD 840 6B0Q /dev/sdb
[3:0:0:0] disk ATA SAMSUNG MZ7KM960 003Q /dev/sdc
... [6:0:0:0] disk JetFlash Transcend 16GB 1100 /dev/sdh
```
In this case, /dev/sdh is my /dev/$USB .
If you wish to turn package inclusion off (on by default), add PACKAGES=0
to the make command.
make PACKAGES=0
Packages include zfs subsystem, and a few related tools.
The bootable kernel and initramfs will be located in /mnt/root/boot. Copy them to the appropriate location for serving using iPXE and http on your machine.
After configuring your system for iPXE boot, with the kernel and initramfs located and available for service via http (much faster than tftp), make sure you add the following boot options to the kernel line.
root=ram rootfstype=ramdisk udev.children-max=1 simplenet=1 verbose console=tty0
You can add options to this as needed.
net.ifnames=0|1
setting this to 0 will disable renaming networks from
eth$N. Setting this to 1 will use new naming scheme. Default is 1.
simplenet=0|1
will remove/re-insert drivers for NICs, loop through all
the network devices, bringing the NIC up, and then looking for a carrier.
Those that have a carrier will be dhcp'ed. Using simplenet=1
is
is the simplest way to bring up a network
net_if=NET
configure the NET device (must come before sub options
below)
net_addr=IP/MASK
set a particular IPv4 IP address/CIDR MASK to the
NET addressbr_name=NAME
create a linux bridge device named NAME (must come
before the sub options below)
br_if=NET
attach NET network device to the NAME bridge
br_addr=IP/MASK
set a particular IPv4 IP address/CIDR MASK to the
NAME bridge. If not included, the bridge NAME will dhcp for an address
For both net_if=NET
and br_name=NAME
, the following suboptions exist
net_dns=IPDNS
creates an nameserver IPDNS
entry in /etc/resolv.conf
net_gw=IPGW
add a network gateway at address IPGW
rootpw=PLAINTXT
set a root password on boot. Not secure, but usable
in an emergency. The image is immutable, but the instance of the image is not.
So you can use this to start up an image instance with a new root password for
running you own tests. Since the instance image will not, by default, attach
any specific durable storage, you can use this as a sandbox for testing.
rootsize=X{G.M,K}
set the tmpfs (if used) root disk size to X with units
as indicated.
disablettyS{0,1,2}=0|1
enable (0) or disable any of the serial consoles (activated by default) from being used for logins.
enablelldpd=0|1
turn LLPD off (0) or on (1). Off by default.
zpoolimport=1
For ZFS builds only, force a zpool import.
ramdisktype=zram
to use a zram (compressed ramdisk block device) rather
than the default tmpfs device. This will create an ext4 file system atop the
/dev/zram0
device, and mount it as your root file system.
ramdisksize=SIZE_IN_GB
to set the ramdisk size to be SIZE_IN_GB number of
GB. So if you wish to use a 5GB ramdisk, use ramdisksize=5
.
image=keep
copies the boot tarball into / on the booted image. This
is useful if you would like to use this to install this (exact) booted system to
physical disk. You will need to install and configure grub-pc after this,
but that is generally a simple process.
Normal kernel boot parameters also apply to the image instance. Due to udev
issues on startup, using udev.children-max=1
is highly recommended. You
may encounter race conditions if you use a number higher than 4. You may turn
on basic deugging using debug
, or set console to physical terminal console=tty0
.
It is possible to break startup at a number of points before, during, and after unpacking the rootfs into the ramdisk.
break=preunpack
will launch a shell before the system creates the ramdisk
debug
on the command line will also return the environment variables
prior to creating the ramdisk
break=postunpack
will launch a shell after the system creates the ramdisk
and unpacks the snapshot tarball.
Exiting these shells should continue startup operation.