Copyright (c) 2014 cozybit Inc. All rights reserved

INTRODUCTION ############

distro11s is a quick and dirty ephemeral roll-your-own linux distribution based on Debian that adds packages and a suitable kernel to play with mesh networking. It is "ephemeral" because all of the work is meant to be pushed upstream regularly. So most folks who want to use mesh will pull their code from upstream, not from us. But we need a coherent way to track and test the latest and greatest. This document explains all the steps needed to build distro11s. For details on the supported target boards, see the README files in the board/ directory.

distro11s is typically deployed uniformly on several identical nodes. Each node needs its own name and IP addresses. To save developers from having to maintain special DHCP configurations for assigning these, distro11s uses a simple host number. The host number is the only board-specific configuration value, and it lives in /etc/distro11s-hostnumber. It is typically set by an installer that increments his own counter every time he successfully installs distro11s on a target. It is perfectly legal for /etc/distro11s-hostnumber to not exist. But when it does exist, it influences various system settings, most notably static IP addresses and the hostname. See distro11s.sample.conf for more details.

SAFETY AND SECURITY ###################

It should be noted that little effort has been made to make distro11s safe and secure. Building it does require super user access for all kinds of unsavory tasks such as formatting file systems, blowing away the file system staging directory, etc. When distro11s wants to do something as the super user, it generally warns you and requires your OK. But it is entirely possible that a poorly or hastily written distro11s config file could result in blowing away your entire root file system. Seriously.

It should also be noted that some features (such as the SSHFS automount) are handy for developers, but have inherent security flaws. That feature in particular adds a key with no password to your authorized_keys file automatically! If you were to enable this feature and distribute the distro11s file system, anybody with that file system and a link to your laptop could log in to your development machine! Seriously! You will be warned at build time about this sort of stuff. But seriously! Be careful.

BINARY RELEASES ###############

If you have a binary release, you must untar it as sudo because it contains a rootfs that has some files that belong to root. Yeah, not so pretty. Sorry. Anyway, at that point, follow the board-specific README for your binary to learn how to deploy it to your hardware.

SETTING UP YOUR DEVELOPMENT SYSTEM ##################################

We build distro11s on Ubuntu 11.04. Here's how:

1. Install the packages that you'll need to build the distro. Note that this list of packages is what's required to build all packages for any conceivable board.

   $ apt-get install python-dev uml-utilities git debootstrap autoconf libtool flex bison qemu

   $ wget -O /usr/share/qemu/pxe-rtl8139.bin \ "http://svn.savannah.gnu.org/viewvc/*checkout*/trunk/pc-bios/pxe-rtl8139.bin?root=qemu" $ wget -O /usr/share/qemu/pxe-e1000.bin \ "http://svn.savannah.gnu.org/viewvc/*checkout*/trunk/pc-bios/pxe-e1000.bin?root=qemu"

   $ sudo apt-get remove gnu-fdisk

2. Create a configuration file for your build. This is just a bunch of bash variables that influence the build. The recommended way to create a config file is to copy distro11s.sample.conf to distro11s.conf and edit it. It's loaded with comments, so read it for more details.

   If you want to support simultaneous builds for different boards, you can have a different config file for each board. Just set the DISTRO11S_CONF environment variable to the config file that you want to use. You may wish to set the DISTRO11S_SRC variables in different config files to keep the per-package builds separate. DISTRO11S_OUT, on the other hand, should co-exist fine for different boards.

FETCHING THE CODE #################

This is a distribution. Accordingly, it is made up of many packages. To fetch all of the packages into your $DISTRO11S_SRC directory, say:

$ ./scripts/fetch.sh

BUILDING THE CODE #################

To build all of the packages, say:

$ ./scripts/build.sh

Note that all targets use debbootstrap to prepare the basic rootfs and must be built as root. You will be warned if/when this happens.

ADDING A NEW SUPPORTED BOARD ############################

Create a directory in board/ with the name of your board. Consider cloning the board directory for a similar target. Note the board//.conf file. It contains board-specific overrides that can be used to tweak the build.

PROVISIONING DISTRO11S ######################

The provisioning.sh script allows you to provision distro11s onto a hard drive. In order to do this, please follow the next steps:

• Get a hard drive and connect it to your dev machine.
• Get its device name possibly from fdisk -l or dmesg (e.g., /dev/sdb)
• Run the next command: $ ./scripts/provisioning.sh -d
• Install the hard drive back on the target

In order to make the installation process easier, a USB Installer can be created using this script. This USB installer will allow you to install distro11s by just plugging the USB key into the target. To create a USB installer, please follow the next steps:

• Connect your USB device to your dev machine (2Gb minimum)
• Get its device name possibly from fdisk -l or dmesg (e.g., /dev/sdb)
• Run the next command: $ ./scripts/provisioning.sh -d -i
• Unplug the USB device and connect it to the TARGET
• Boot from the USB, and the installation process will begin once is booted
• Once distro11s is installed, the target will reboot, loading distro11s

This USB installer approach is convenient when installing distro11s on many machines. Note, however, that data throughput between the development machine and a USB drive varies dramatically. In some cases, copying the half GB or so of data may take 30 minutes or even longer. Consider using a higher-performance external USB harddrive.

TIPS AND TRICKS ###############

-- Many parts of the distro11s build system use stamp files in the out//stamps/ directory. This way, scripts can be run multiple times without repeating very time-consuming steps that might be unnecessary. To force various parts of the build to execute again, delete the appropriate stamp file. Inspect the contents in the stamps directory for more details.

-- The stamp system is not very sophisticated at this time. It's mostly used for the release procedure, which is a complete clean build. As you develop, you may find it easier to hop around the DISTRO11S_SRC directory manually invoking make/make install (or whatever). Alternatively, you can invoke the build script with FORCE_BUILD=1. For example, if you want to execute ALL of the build steps for the kernel (including coping the config over!) you can say:

FORCE_BUILD=1 ./scripts/kernel.sh

-- Copmile / test cycle in qemu:

1. make a change
2. rebuild your kernel, 'FORCE_BUILD=1 ./scripts/kernel.sh'
3. push your new modules to your running qemu, './scripts/push -u root@${DISTRO11S_HOST_IP}'
4. test.

-- Avoid using sudo. But if you must (e.g., mount, debootstrap, etc.), use the root_check function before invoking sudo. This alerts the user.

-- If you're adding a new build script, try to make it repeatable. That is, if I blow away all of the stamp files and invoke your build script, it should do the right thing.

-- Make a minor change to a package and just want to push your changes to a node that you already provisioned? Try this:

$ FORCE_BUILD=1 ./scripts/.sh $ ./scripts/push.sh root@ipaddr-of-target

-- If you want to save yourself the trouble of entering your password every time you ./scripts/push.sh, you must set up the root user _on_your_devmachine to have a password-less ssh key, and you must ssh-copy-id that key over to each device to which you wish to push.sh. The reason is that push.sh rsyncs as sudo.

-- Got some operation you want to perform on each node? Consider using the foreach.sh script. You'll have to set the DISTRO11S_NUM_NODES variable in your config file, then you can say something like:

./scripts/foreach.sh './scripts/push.sh [-u root@$HOSTNAME.local]'

or

./scripts/foreach.sh 'ssh root@${HOSTNAME}.local ""'

Your little bit in single quotes will get executed once for each node. The vars that are set by the foreach.sh script (currently) include:

HOSTNUM: The hostnumber

IP: The IP address of the node formed by adding his hostnumber to DISTRO11S_STATIC_IP.

HOSTNAME: The hostname of the node formed by adding his hostnumber to DISTRO11S_HOSTNAME.

Note: you can customize the range of nodes you would like to sync by setting DISTRO11S_START_NODE and DISTRO11S_END_NODE in your distro11s.conf.

Note: to connect to qemu with gdb, while inside gdb type: (gdb) target remote ${DISTRO11S_HOST_IP}:1234

-- If you are working on an older machine that does not support hardware virtualization, you will have to disable kvm:

diff --git a/board/qemu/qemu.sh b/board/qemu/qemu.sh index 020daf1..0b7f730 100755 --- a/board/qemu/qemu.sh +++ b/board/qemu/qemu.sh @@ -27,7 +27,7 @@ ${QEMU} -kernel ${KERNEL} -hda ${ROOTFS} \ -append "root=/dev/sda combined_mode=ide console=ttyS0" \ -nographic -net nic,model=e1000 -net tap,ifname=${IFNAME},script=no \ -gdb tcp::1234 \

• -enable-kvm -smp 2
• -no-kvm -smp 2