[enhancement] CLI development

[ukkopahis]

I'm working on a CLI interface and a proper back-end API for stack modification and upgrades.

Design goals:

• All state is stored into docker-compose.yml and .env
  • no services/ folder is needed
  • overrides not needed for standard use
• Python module for stack modification - used by both CLI and menu (and any future web-UI)
• By-hand usage using command line-tools possible.
• Template specification
  • No service specific python code in .templates/
  • Menu/CLI configuration options are inferred from template variable patterns
  • All previous overrides continue working, not requiring any migration.
• Use of sane defaults to reduce required user configurations to a minimum. Too many options will just introduce confusion to new users, and make it easier to break things.

Template specification (service.yml)

The service.yml-templates are currently yaml-snippets, with a custom variable replacement scheme (%var%). Docker compose files natively allow for variables using the variable substitution syntax, e.g. ${VARIABLE:?error message} or ${VARIABLE:-default_value}. The custom notation will be migrated to use compose variables.

Thus a service.yml-file is defined as:

• A valid yaml document with an initial two space indent
  • Each root defining a docker-service
• Docker Compose variables starting with IOTSTACK_ have a special meanings in the menu.
• service.yml snippets can also be hand-concatenated into a docker-compose.yml and the required variables defined in .env. No menu-use is strictly necessary.

The menu should store user-entered variable values into .env, where they are read from when re-running menu to show the previously made choices.

Special templating variables

service.yml variable	assumed meaning
IOTSTACK_SERVICE_PASSWORD or IOTSTACK_SERVICE_INITIAL_PASSWORD	generate random password or use an user defined password
IOTSTACK_SERVICE_VARNAME_CHECKLIST	space-separated default values user may deselect
IOTSTACK_SERVICE_VARNAME_CHECKLIST_UNSELECTED	per default unselected values available to be selected
IOTSTACK_SERVICE_VARNAME_BOOL	boolean choice, values 'true' or 'false'
IOTSTACK_SERVICE_VARNAME_INT	integer value choice
IOTSTACK_SERVICE_VARNAME	string value choice
# comment on the same yaml-line as a variable	provide UI description for the value choice
IOTSTACK_TZ	Use /etc/timezone, but detect if TZ is defined in shell-env or .env
IOTSTACK_HOSTNAME	replaced using the hostname
IOTSTACK_LAN_IP	constant resolving to the IP of eth0 or wlan0, print warning if this is an ip obtained by DHCP
IOTSTACK_LAN_MAC	constant resolving to the MAC address of eth0 or wlan0
IOTSTACK_UID	user id
IOTSTACK_GID	group id

Example (combining many different service use-cases):

environment:
  - TZ=${IOTSTACK_TZ:-Etc/UTC}
  - WEBPASSWD=${IOTSTACK_PIHOLE_INITIAL_PASSWORD:-IOtSt4ckP1Hol3}
  - TOKEN=${IOTSTACK_DUCKDNS_TOKEN:?token from duckdns.org required}
  - PLUGINS=${IOTSTACK_NODERED_PLUGINS_CHECKLIST:-node-pi-gpiod contrib-influxdb contrib-boolean-logic node-rbe configurable-ping dashboard} # Addons available for NodeRed
  - DUMMY_PLUGINS_NON_DEFAULT=${IOTSTACK_NODERED_PLUGINS_CHECKLIST_UNSELECTED:-ode-openweathermap contrib-discord node-email ...} # This defines a dummy variable in order to add the list of unselected items.
ports:
  - ${IOTSTACK_HEIMDALL_HTTP_INT:-8880}:80 # HTTP port
  - ${IOTSTACK_HEIMDALL_HTTPS_INT:-8883}:8080 # HTTPS port
entrypoint:
  - ash
  - -c
  - |
    cd /usr/src/node-red
    for a in $$PLUGINS ; do npm install --save  node-red-$$a ; done
    npm start --cache /data/.npm -- --userDir /data

In this example menu would only require user to enter the duckdns_token, rest of the variables would be their default values. Note: the entrypoint is an example how to implement dynamic functionality based on variable values.

Manual use without menu

Additionally operation without using the menu is possible (when restoring proper indentation to service.yml-files and appending "services:" to env.yml/docker-compose-base.yml) by e.g.:

$ cat .templates/docker-compose-base.yml .templates/pihole/service.yml > docker-compose.yml
$ echo "IOTSTACK_PIHOLE_INITIAL_PASSWORD=trulysecret" >> .env
$ docker-compose up

Development

Current WIP version: template.py

Feel free to take a look and give feedback or comment if something doesn't look right.

Tasks:

• [x] use virtualenv to avoid polluting the host with pip dependencies
• [x] operation: check templates
• [ ] convert templates to use Docker compose variable substitution syntax
• [x] operation: list
• [ ] operation: add (add template name as metadata into a comment when a single template adds multiple services)
  • [ ] compose file structured 'networks:' before 'services:'
• [ ] argument: password replacement
• [ ] argument: variable replacement (add metadata into a comment to enable preservation when updating)
• [ ] operation: delete
• [ ] operation: recreate
• [ ] operation: update
• [ ] detect missing devices (/dev/xxx) - if a service references a non-existing device, emit a warning about it (possible causes and solutions) and comment out that line. (Leaving the line would prevent docker from starting the service)
• [ ] eliminate need for template specific python code by using the replaceable variables in service.yml
• [ ] update menu to use the same backing python code for all stack operations.
• [ ] update docs

Container changes/fixes (that are much easier):

• [ ] fix conflicting ports - all services can be installed at the same time using their default configuration. (with the exception of udp/53 used by both pihole and adguardhome)
• [ ] change node-red to do addons installation using an entrypoint-definition in docker-compose (example) (And thus not need the Dockerfile.template generated image)
  • Will make the docker-compose definitions for the service a bit longer, but it's worth it in order to reduce current layers of complexity. Also makes further customization easier, by having everything explicitly visible in the docker-compose.yml-file.
• Homebridge on 64bit should use tag :no-avahi-arm64v8
• Telegraf 32/64-bit dependent version selection
• Show Influx v2.x service only when on a 64 bit kernel supporting it.

Definition of Done:

• Complete unit and integration tests for the API
• No pylint warnings

Additional ideas for the future:

• Rewrite 'build stack' menu to provide all functionality available in the CLI. Migrate to urwid, as using blessed or curses is a bit too low-level.
• script to create a stack with all available services and check that everything starts OK and nothing ends up in a restart loop
• add github hook/actions to run python tests&linting and to run template validity checks (for each push and pull-request)
• make IOTstack installable and updatable as a python package module: "pip3 install git+https://github.com/SensorsIOT/iotstack.git"
• aliases/"terminal.sh" scripts can use CLI to get login passwords automatically
• integrations/additions that are dependent on other services being chosen, e.g. samba shares for Octoprint printables and qbittorrent/transmission downloads.