Using the docker module causes build to fail

[David00]

I'm building on a Pi 4B running Ubuntu and have followed these build instructions very closely. I'm just simply trying to implement the docker module in an otherwise stock build, and the build fails.

I can build a stock image by following the linked instructions, but once I edit the config file to add the docker module, the build fails. I do not have any custom start/end chroot scripts.

I have attached the build log to this issue, but the specific failure message is no space left on device: '/tmp/pip-wheel-4ppioxub' after installing the Python cryptography package.

Here's what I'm using to create a fresh distro environment:

cd ~/CustomPiOS/src
./make_custom_pi_os -g ~/new_distro
cd ~/new_distro/src
sudo modprobe loop
sudo bash -x ./build_dist

This works just fine with no customizations to any of the files in ~/new_distro/. However, when I add the docker module to the ~/new_distro/src/config file, and try to build again, it fails. Here is the config with the docker module included:

export DIST_NAME=new_distro
export DIST_VERSION=0.1.0
export MODULES="base(network,docker(new_distro))"

I can't imagine my disk space is the problem, but my Pi is using a 500GB NVMe boot drive via USB 3.0. Here's my df:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           776M  4.7M  772M   1% /run
/dev/sda2       459G   14G  427G   3% /
tmpfs           3.8G     0  3.8G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda1       253M  141M  112M  56% /boot/firmware
tmpfs           776M  120K  776M   1% /run/user/126
tmpfs           776M  100K  776M   1% /run/user/1000
overlay         459G   14G  427G   3% /var/lib/docker/overlay2/5c654b1d1217b4157504c5eee704a4700244ef29d70a7de9df26684b7bd746f3/merged
shm              64M     0   64M   0% /var/lib/docker/containers/5113d7c265783e278b7a11f8c4a640249f3238065b7a17c48856846b56eceb4f/mounts/shm

I have also tried building with docker-compose and the same failure occurs. I am about to try this in a Ubuntu VM on another system of mine since I suspect something strange might be going on surrounding the Pi and its USB boot drive. Build Failed Log.txt

[guysoft]

Hey, This is because the docker module now also needs to build cryptography module and other due to PEP 517. That causes the image to run out of space when building. So the no space left of device is the space inside the image.

The solution is simple and I tested it, add BASE_IMAGE_ENLARGEROOT and BASE_IMAGE_RESIZEROOT in the config, it should extend the size of the image during the build, and then shrink it back again when its done building.

export DIST_NAME=new_distro
export DIST_VERSION=0.1.0
export MODULES="base(network,docker(new_distro))"

export BASE_IMAGE_ENLARGEROOT=2000
export BASE_IMAGE_RESIZEROOT=200

They are part of the base module and are documented here: https://github.com/guysoft/CustomPiOS/blob/devel/src/modules/base/config#L53

Since the docker module is becoming more useful it might be a good reason to change the default value for BASE_IMAGE_ENLARGEROOT so people don't have to set it up explicitly.

[David00]

@guysoft Thanks so much! That cleared things up. I have another question about building images with the docker module, if you don't mind.

I intend to pull a couple of docker images on boot. I've put a docker-compose.yml file in filesystem/boot/docker-compose/arbitrary_folder_name/, and added a user-data file to filesystem/boot/, which I believe runs the /usr/bin/start_docker_compose script.

I've been investigating why docker-compose isn't pulling the images on the first boot (I have a physical network connection on boot), and when running the /usr/bin/start_docker_compose script manually, I see there's an import error raised from /usr/local/bin/docker-compose: ImportError: cannot import name 'ContextAPI' from 'docker' (/usr/local/lib/python3.7/dist-packages/docker/__init__.py)

I don't see you specifically installing the docker library in any of your setup scripts in PhotoPrismPi, so I'm not sure what I'm missing. I am currently trying a build where I've added python3 -m pip install docker to the start_chroot_script to see if that helps, but I wanted to see if you had any ideas.

Update: Hmm... docker and docker-compose are already installed and are the latest versions. I will dive deeper into this later and attempt to find/fix the issue.

[guysoft]

1. Looks like a docker bug.
2. It might be that there is no docker-compose.yml file in the folder. Can you show the the output of :

   ls /boot/docker-compose/arbitrary_folder_name/

   on the running system? Because the NVME boot might be changing the mount path of `/boot, which I cant see mounted on the Pi. It should be something like this on the df command on a Pi using an SD card:

   /dev/mmcblk0p1  253M   54M  199M  22% /boot

Also providing the docker-compose.yml might help to confirm it has no typo in it.

[David00]

I agree, it seems like a bug with docker-compose. Here is the content of /boot/docker-compose/mydockerfolder on a Pi running the custom built image (I previously said arbritary_folder_name just to illustrate it wasn't named after the distro name) :

pi@newdistro:~ $ ls /boot/docker-compose/mydockerfolder/
docker-compose.yml

The contents of the docker-compose.yml file are:

version: "3.9"
services:

  influxdb:
    image: influxdb:1.8.3
    volumes:
     - /opt/influxdb:/var/lib/influxdb
    ports:
      - "8086:8086"
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s

  grafana:
    image: grafana/grafana:7.3.7
    ports:
     - "3000:3000"
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s

Here's the output of what happens when I manually run the /usr/bin/start_docker_compose script with bash on the custom image:

pi@newdistro:~ $ bash /usr/bin/start_docker_compose
/boot/docker-compose/mydockerfolder ~
Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 5, in <module>
    from compose.cli.main import main
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 29, in <module>
    from ..metrics.decorator import metrics
  File "/usr/local/lib/python3.7/dist-packages/compose/metrics/decorator.py", line 3, in <module>
    from compose.metrics.client import MetricsCommand
  File "/usr/local/lib/python3.7/dist-packages/compose/metrics/client.py", line 5, in <module>
    from docker import ContextAPI
ImportError: cannot import name 'ContextAPI' from 'docker' (/usr/local/lib/python3.7/dist-packages/docker/__init__.py)

Here's the /boot partition on a Pi running the image via an SD card: /dev/mmcblk0p1 258095 53470 204626 21% /boot

[guysoft]

Looks ok, except you are missing a "deploy" block. Should be:

version: "3.9"
services:

  influxdb:
    image: influxdb:1.8.3
    volumes:
     - /opt/influxdb:/var/lib/influxdb
    ports:
      - "8086:8086"
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s

  grafana:
    image: grafana/grafana:7.3.7
    ports:
     - "3000:3000"
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s

Note there are two deploy blocks, here now just one

[David00]

Great catch! Thank you. I did some digging and found out it's the docker-py package that breaks docker-compose. (See issue here).

I removed the python docker-py library manually from a Pi running my latest build, and that allowed docker-compose -v without the import error. Then, I tried to manually run the /usr/bin/start_docker_compose and found out that it doesn't like compose version 3.9 yaml files, so I dropped the version to 3.3, and it was able to successfully pulled the docker images down.

I don't yet understand how docker-py is making its way into my build - I can't find it specifically listed in any of the build config files, but I am trying a new build now where I've added pip3 uninstall docker-py to the docker module's start_chroot_script.

[David00]

Hah, I am fighting against myself, here. docker-py is coming from my requirements.txt file and is used in my application.

I'll have to find another workaround. I will close this issue now. Thanks so much for the help!

[David00]

@guysoft, I have another question that is somewhat related to this issue with regards to resizing the image. I thought about opening a new issue for this but I want to make sure I'm not doing something stupid.

So, two issues, potentially related to each other:

• Edit: Ignore this... I was doing something stupid and I have answered this part in my next comment below... Final image size is much larger than expected: When my build completes, I get a .img file in distro-name/src/workspace/ that is 3.3GB in size.
• It's named the same as the stock image, and I don't see a .zip anywhere. The build script seems to be ignoring my custom distro name variable and it's not creating a .zip.

I saw issue #36 about resizing the image, and I already have the resize environment variables specified in my distro-name/src/config file thanks to your recommendation above.

Here is my config file in distro-name/src/config (I'm not sure if the POWERMON_ variables go in this config or in modules/powermon/config but I don't think it matters):

export DIST_NAME=powermon
export POWERMON_DIST_NAME=powermon
export DIST_VERSION=0.1.0
export POWERMON_DIST_VERSION=0.1.0
export MODULES="base(network,docker,powermon)"
[ -n "$POWERMON_REPO_SHIP" ] || POWERMON_REPO_SHIP=https://github.com/David00/rpi-power-monitor.git
export BASE_IMAGE_ENLARGEROOT=2000
export BASE_IMAGE_RESIZEROOT=200
export BASE_RELEASE_COMPRESS=yes
export BASE_RELEASE_IMG_NAME=Raspberry-Pi-OS-Lite_$DIST_NAME-$DIST_VERSION
export BASE_RELEASE_ZIP_NAME=Raspberry-Pi-OS-Lite_$DIST_NAME-$DIST_VERSION.zip

I have looked into the issue a little bit, and I see the .zip should be created by the src/release file. However, I can't figure out how this release script is called, and it seems that this is never executed.

In my build.log, I see:

Resizing partition 2 on 2020-02-13-raspbian-buster-lite.img to minimal size + 200 MB

So, it leads me to believe that the resize part of the script is incorrectly determining the minimal size.

As a test, I reduced BASE_IMAGE_ENLARGEROOT to 500 instead of 2000 and reran my build, and the .img file size was reduced from 3.3GB to 2.5GB.

So, it seems that the minimal size is based off of whatever is set in ENLARGEROOT, and not in the actual disk usage size. The output from my start_chroot_script says my apt-get install applications will use a total of ~371 MB of disk space. So, I don't think my image should be any more than 1 GB in size before compression.

So, in summary:

1. How can I get the build script to give me a .zip?
2. Why isn't the image size being reduced as expected?

[David00]

I did some digging and found comments in other issues that suggest the .zip is only created when running the src/release script, but I haven't been able to find anything in the docs about using the release script.

Nonetheless, I've tried copying src/release to my build directory, and running it there. I'm getting some issues about not having CUSTOM_PI_OS_PATH set, but it is...

david@pi-workshop:~/custompios-builds/powermon/src$ sudo ./release
/home/david/custompios-builds/powermon/src/workspace /home/david/custompios-builds/powermon/src
Error: you must have ${CUSTOM_PI_OS_PATH} set

david@pi-workshop:~/custompios-builds/powermon/src$ echo $CUSTOM_PI_OS_PATH
/home/david/CustomPiOS/src

So, I'm on track to figuring out how to use the release script create the zip... but figuring out why the image file isn't being reduced to what I think it should be is still open.

I think I'll just create the zip manually for now until you're able to assist.

Edit:

zipping manually did one heck of a job, taking my image from 2.5GB down to 782MB. I was not expecting that level of compression, and that is exactly what I expect the image size to be. I was seeing the zipped size of the stock Raspberry Pi OS at 434MB and wondering why my .img file was so much larger. I now see that I was comparing apples to oranges, and that the stock image file when unzipped is around 1.8GB.