Dockerized Mycodo

[kizniche]

This is my experiment with using docker and docker-compose to run Mycodo. Thus far I successfully have Mycodo running fully in docker containers. This includes the Flask/Gunicorn frontend, Nginx, Influxdb, and the Mycodo daemon (additionally with Grafana and Telegraf for further data acquisition/presentation). This issue can serve as a discussion thread and a way I can keep users updated on the progress and where this is going. For current information about setup/building, see the docker/README.md.

Current state (check means working):

• [x] All containers start up and communicate (daemon, flask, nginx, influxdb)
• [x] Internal sensors (CPU load, disk space, etc.)
• [x] Device sensors (i.e. /dev/ttyUSB0)
• [x] Telegraf and Grafana
• [ ] Outputs
• [ ] Upgrade
• [ ] Backup/restore

More will be added as they are found

[jonny-sexton]

Brilliant, thanks for the speedy response! I retried the steps outlined above on the docker-01 branch, and the orginial issue is solved. However I now have a new error message:

ERROR: Service 'mycodo_daemon' failed to build: The command '/bin/sh -c /home/mycodo/mycodo/scripts/upgrade_commands.sh install-docker' returned a non-zero code: 100

I had a quick look, but tbh I don't understand where the issue lies.

[jonny-sexton]

unless we can figure out a way to detect this automatically.

You can detect which chip architecture a linux machine has with the arch command. On the pi zero I get the following:

armv6l

On the pi 4 you should get

aarch64

I believe.

[kizniche]

I just committed (and merged into master) my final changes for Docker. I haven't tested on a Pi Zero.

I now have a new error message: ERROR: Service 'mycodo_daemon' failed to build...

Is this all there was in the log? No other errors?

[jonny-sexton]

OK, I will try again then with the master branch. I ran the docker-compose up command with the --verbose option and here are the last few lines of code before the error appears:

compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0xb4849530>
compose.cli.verbose_proxy.proxy_callable: docker close <- ()
compose.cli.verbose_proxy.proxy_callable: docker close -> None
compose.service.build: Building mycodo_daemon
compose.cli.verbose_proxy.proxy_callable: docker build <- (path='/home/Jonny/Mycodo', tag='app', rm=True, forcerm=False, pull=False, nocache=False, dockerfile='docker/Dockerfile', cache_from=None, labels=None, buildargs={}, network_mode=None, target=None, shmsize=None, extra_hosts=None, container_limits={'memory': None}, gzip=False, isolation=None, platform=None)
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
docker.utils.config.find_config_file: Trying paths: ['/home/Jonny/.docker/config.json', '/home/Jonny/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.api.build._set_auth_headers: Sending auth config ()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/build?t=app&q=False&nocache=False&rm=True&forcerm=False&pull=False&dockerfile=docker%2FDockerfile HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0xb48e4e28>
ERROR: compose.cli.main.main: Service 'mycodo_daemon' failed to build: The command '/bin/sh -c /home/mycodo/mycodo/scripts/upgrade_commands.sh install-docker' returned a non-zero code: 100

[kizniche]

This command starts with printf, so for that to not be in the logs indicates an unusual issue.

https://github.com/kizniche/Mycodo/blob/2c200363c85c8c3988bdc8c576faf90c85419b6e/mycodo/scripts/upgrade_commands.sh#L711-L715

[jonny-sexton]

I just tested with the main branch again, but still no dice:

Building mycodo_influxdb
Building mycodo_flask
ERROR: Service 'mycodo_flask' failed to build: The command '/bin/sh -c /home/mycodo/mycodo/scripts/upgrade_commands.sh install-docker' returned a non-zero code: 100

This command starts with printf, so for that to not be in the logs indicates an unusual issue.

Yes that's what I thought. I'll take a closer look when I have more time, will probably be this weekend though. I'll update you if I find anything.

[kizniche]

I just did a full test on a freshly installed Ubuntu 20.04 virtual machine. I changed the Linux image used and updated the build commands. I was able to build and run without issue. It also appears "docker compose" changed how output is displayed in the terminal from "docker-compose", which is much less verbose. I'm looking to see if it's possible to get the old behavior back. --verbose doesn't seem to have any effect, which I believe is a bug.

[jonny-sexton]

Hi @kizniche, sorry for the late reply. I still couldn't get this to work, and I have decided to write my own code for my needs.

However I think the crux of the issue was that my raspberry pi zero w (1st generation) supported 32-bit OS only, and Docker does not support 32 bit OS (I had to install Docker using unofficial 3rd party software: https://gitlab.com/docker-32bit). This caused all kinds of issues trying to fix it, so I think you should maybe add in the README that 32-bit raspberry pis are not supported for the docker installation of Mycodo.

[nargetdev]

architecture heterogeneity can be addressed with buildx as follows..

#! /bin/bash
docker buildx build \
            . \
            --push \
            --platform linux/arm64,linux/arm/v7,linux/arm/v6 \
            --tag nargetdev/infinilapse:1.1.2

[nargetdev]

@nargetdev could you please provide some more information about building/running on kubernetes? I'd like to explore this option more, but my experience is virtually all with Docker, so I could use some help getting this going and want to avoid fumbling around and hitting common issues that can be avoided with some guidance. Thanks.

@kizniche right now my workflow is using kompose tool to map the docker-compose.yml to kubernetes manifests. It mostly accomplishes an equivalent semantics of the services, containers, volumes, networks, etc. in k8s parlance.. but not 100% The main difference is how the volumes are initialized by k8s.

Here's my kompose output directory (nothing special .. just vanilla output from kompose convert -o kompose) https://github.com/nargetdev/Mycodo/tree/master/kompose

On the one hand docker compose chooses to initialize a new volume implicitly on creation by copying over the contents at a given path from the container to initialize the volume contents.. however if the volume already had some contents at all (i.e. hi.txt) then the behavior is for the volume to clobber the contents within the container at the provided path. Kubernetes I guess views this as a liability and an edge case because it requires clients, scripts, etc to explicitly delete/recreate volumes for deterministic behavior. For this reason Kubernetes maintains only the latter behavior: always clobber at path, even with an empty volume.

Insofar as this concerns us we need to some method to bootstrap the volumes on first launch.

P.S. I'm going to make a separate issue but i'm just starting to look into the upgrade/restore function in container context.

[nargetdev]

Actually i'll consolidate here regarding upgrade backup/restore since it's in the checklist above..

The first thing I'm facing down right off the batt is that the Docker containers are all set up with user mycodo and root directory stub mycodo.. as opposed to bare metal whatever_user and root directory stub Mycodo

Many of the scripts are hard coded with Mycodo.. i've started to convert them in my branch.

To illustrate:

on baremetal

MYCODO_ROOT=$( cd -P /var/mycodo-root && pwd -P )
echo $MYCODO_ROOT ==> /home/soma/Mycodo

in container

MYCODO_ROOT=$( cd -P /var/mycodo-root && pwd -P )
echo $MYCODO_ROOT ==> /home/mycodo

Is this a mistake or is there a reason for the inconsistency?

===

Secondly, perhaps more fundamentally, baremetal instance plays with systemd to stop the processes, but it's not set up this way for docker.

systemctl list-units --type=service ==> shows stuff for baremetal ==> error within docker

From a 20,000' view the "docker" way of handling an upgrade/restore would be to rip down the container, rehydrate the stateful filesystem via external process (the volumes), and then boot the containers. This could be orchestrated nicely with k8s control plane. The "normal" way of doing it is where the system inside the container is none the wiser, stops it's process.. with this said my preliminary attempts at docker top mycodo_daemon ps aux and kill -9 <pid> have all failed at stoping the command specified by docker (which makes sense.. that command is the raison d'etre of the container so i'm not surprised it's hard/impossible to stop from that valence).

Let me know your thoughts on the two avenues / sanity check my logic @kizniche

[nargetdev]

Great success!

Mycodo on k8s..

https://github.com/kizniche/Mycodo/pull/1263

[kizniche]

Is this a mistake or is there a reason for the inconsistency?

I found it easier to work with, but there's no reason other than that. The symlink mycodo-root is what I've been using to identify the root wherever it may be, since different users could be installing it in their home directory. Really, it should be installed in a more defined directory like /opt or /usr, for example, but this is difficult to change after having it at a different location for so many users.

I haven't been able to figure out a reliable way to perform an upgrade in docker. I agree this needs to be done external of the containers, with some sort of script. Discourse is one such project I use that utilizes docker and an external upgrade system with scripts, so perhaps we can look to that for inspiration.

[nargetdev]

Off the bat there are no systemd services to stop. These are rather the containers themselves, which should be ephemeral (able to be destroyed without notice)

An update could Docker "commit" the changes (I.e. dependencies installed) and use the newly committed image, but this seems messy.. The dependencies could persist in a volume as is the case currently.. but this is prone to corruption unless the volumes are incrementally backed up.

Finally a "fat" container could be built with all possible dependencies preinstalled.

What else is stateful outside of the running version of the sources?

mycodo.db

.. what else?

If you could help me just enumerate a bullet point list of stateful components so I can start my study? Thank you kindly

[samuk]

I've run into an issue where libcamera-apps dependency won't install

But following that:

[kizniche]

This is because Raspberry Pi OS is not the operating system installed in the Mycodo container. For that package, you need to use a Raspberry Pi OS variant to have access to their apt package source. You could try adding the package source to the current OS, but YMMV. Are you running Mycodo in Docker on actual Pi hardware? If so, why use Docker instead of installing on the actual hardware?

If you look in the Docker file, you'll see "debian:stable-slim" is used.

You will need to duplicate the Function and create a new one that uses a more generic package for libcamera than the Raspberry Pi OS-specific libcamera-apps, if you want to use a libcamera Function on an OS other than Raspberry Pi OS.

[samuk]

Thanks, will give that a go.

Yes running in Docker on a Pi, just to test it out. Interested in running it under https://github.com/balena-os

The reason being that I could then swap the underlying hardware for any of their supported devices https://www.balena.io/os#download-os including this one https://www.compulab.com/products/iot-gateways/iot-gate-imx8plus-industrial-arm-iot-gateway/ which I like the look of.

[samuk]

Just adding some notes in case this is useful to someone else later (including future-me)

Using the https://www.portainer.io/ webui

I accessed the terminal for the mycodo_flask container.

apt-install nano

nano /etc/apt/sources.list paste: deb http://archive.raspberrypi.org/debian/ bookworm main

Did a wget http://archive.raspberrypi.org/debian/raspberrypi.gpg.key apt-key add raspberrypi.gpg.key apt-get update apt-get upgrade I can then run the install process in the Mycodo WebUI.

I haven't yet successfully captured an image with it.

apt install sudo apt-get install v4l-utils

root@Mycodo:/home/mycodo/mycodo# v4l2-ctl --list-devices bcm2835-codec-decode (platform:bcm2835-codec): /dev/video10 /dev/video11 /dev/video12 /dev/video18 /dev/video31 /dev/media3

bcm2835-isp (platform:bcm2835-isp): /dev/video13 /dev/video14 /dev/video15 /dev/video16 /dev/video20 /dev/video21 /dev/video22 /dev/video23 /dev/media1 /dev/media2

unicam (platform:fe801000.csi): /dev/video0 /dev/video1 /dev/media0

rpivid (platform:rpivid): /dev/video19 /dev/media4

HiCamera: UVC Camera (usb-0000:01:00.0-1.3): /dev/video2 /dev/video3 /dev/media5

oot@Mycodo:/home/mycodo/mycodo# ls /dev/video /dev/video0 /dev/video10 /dev/video12 /dev/video14 /dev/video16 /dev/video19 /dev/video21 /dev/video23 /dev/video1 /dev/video11 /dev/video13 /dev/video15 /dev/video18 /dev/video20 /dev/video22 /dev/video31 root@Mycodo:/home/mycodo/mycodo# ls /dev/video /dev/video0 /dev/video10 /dev/video12 /dev/video14 /dev/video16 /dev/video19 /dev/video20 /dev/video22 /dev/video3 /dev/video1 /dev/video11 /dev/video13 /dev/video15 /dev/video18 /dev/video2 /dev/video21 /dev/video23 /dev/video31

I can apt install raspi-config

But there is no camera option

root@Mycodo:/home/mycodo/mycodo# v4l2-ctl --all Driver Info: Driver name : unicam Card type : unicam Bus info : platform:fe801000.csi Driver version : 6.1.63 Capabilities : 0xa5a00001 Video Capture Metadata Capture Read/Write Streaming Extended Pix Format Device Capabilities Device Caps : 0x25200001 Video Capture Read/Write Streaming Extended Pix Format Media Driver Info: Driver name : unicam Model : unicam Serial : Bus info : platform:fe801000.csi Media version : 6.1.63 Hardware revision: 0x00000000 (0) Driver version : 6.1.63 Interface Info: ID : 0x03000006 Type : V4L Video Entity Info: ID : 0x00000004 (4) Name : unicam-image Function : V4L2 I/O Flags : default Pad 0x01000005 : 0: Sink Link 0x02000008: from remote pad 0x1000002 of entity 'imx219 10-0010' (Camera Sensor): Data, Enabled, Immutable Priority: 2 Video input : 0 (unicam-image: ok) Format Video Capture: Width/Height : 640/480 Pixel Format : 'Y12 ' (12-bit Greyscale) Field : None Bytes per Line : 1280 Size Image : 614400 Colorspace : Raw Transfer Function : None YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : root@Mycodo:/home/mycodo/mycodo#

This looks potentially useful https://www.losant.com/blog/how-to-access-the-raspberry-pi-camera-in-docker Libcamera commands https://docs.arducam.com/Raspberry-Pi-Camera/Native-camera/Libcamera-User-Guide/ WIP libcamera OpenCV code https://github.com/opencv/opencv/issues/21653#issuecomment-1876971129