Feature: docker-compose.override.yml

[twtg-martijn-bakker]

This PR introduces a few new elements:

1. docker-show-config.sh - outputs the aggregated docker-compose YAML. Very useful during customization.
2. Support for an optional docker-compose.override.yml in the root of the repository.

In docker-compose.override.yml you can add additional services, or override parameters from existing ones.

In addition to the new additional override arguments, the monitoring arguments are now included in all scripts. This prevents errors while running these scripts when there are overrides for prometheus and/or grafana.

Example

The parameters restart and scale are set to always and 20, respectively by default.

$ cat docker-compose.override.yml
version: '2.2'

services:
  tb-js-executor:
    restart: unless-stopped
    scale: 2
$ ./docker-show-config.sh | yq -Y '.services["tb-js-executor"]'
depends_on:
  kafka:
    condition: service_started
environment:
  DOCKER_MODE: "true"
  LOGGER_FILENAME: tb-js-executor-%DATE%.log
  LOGGER_LEVEL: info
  LOG_FOLDER: logs
  REMOTE_JS_EVAL_REQUEST_TOPIC: js_eval.requests
  SCRIPT_BODY_TRACE_FREQUENCY: '1000'
  TB_KAFKA_SERVERS: kafka:9092
  TB_QUEUE_TYPE: kafka
image: store/thingsboard/tb-pe-js-executor:3.3.2PE
restart: unless-stopped
scale: 2

yq is (and uses) jq, but for yaml. Install with python3 -mpip install yq.

[ashvayka]

@volodymyr-babak please review this and merge if ok

[volodymyr-babak]

@twtg-martijn-bakker @ashvayka
at the moment docker-compose files are duplicated into different folders, based on deployment type - monolith, basic and advanced. For instance:

• monolith/docker-compose.prometheus-grafana.yml
• basic/docker-compose.prometheus-grafana.yml
• advanced/docker-compose.prometheus-grafana.yml

This pull request approach is to use the single docker-compose.override.yml file in the root folder and use it for any deployment type: ADDITIONAL_COMPOSE_OVERRIDE_ARGS="-f ../docker-compose.override.yml"

I would suggest using some common approaches - whether docker-compose.override.yml files are created independently inside a specific deployment type folder, e.g.:

• monolith/docker-compose.override.yml
• basic/docker-compose.override.yml
• advanced/docker-compose.override.yml

Or we should do one more round of refactoring and move duplicate docker-compose files to the upper level - to the root folder and then use docker-compose.override.yml file from the root folder as well.

@twtg-martijn-bakker @ashvayka please provide your thoughts on this.

[twtg-martijn-bakker]

Sorry for the late response, I had completely missed your reply.

Personally, I would love to see some refactoring happening here as in my opinion the current setup is not very intuitive to work with.

I have been using docker-compose -f file1.yaml -f file2.yaml config --no-interpolate > docker-compose.yml a lot. This mechanism generates a single, coherent docker-compose.yml from multiple fragments and env files.

It would be nice if this mechanism could be applied here. It would look something like:

user@host:~$ git clone -b "release-x.y.z" <this-repo-url> ./thingsboard-pe
user@host:~$ cd thingsboard-pe
user@host:~/thingsboard-pe$ ./init-tbpe.sh --setup basic --database hybrid --with-monitoring --load-demo-data

The init-tpbe.sh script would take care of all the things, like:

• generating docker-compose.yml
• generating or updating .env
• creating local directory structure, setting permissions etc
• pulling containers

The generated docker-compose.yml and .env files would be placed in the root path of the repository.

This would improve upon the current situation in several ways:

• The repo root will become the project-directory for docker-compose
• the project-name will now be thingsboard-pe
• the docker-compose command can now be used without multiple -f options

For upgrading, a similar script should be created.

Lastly, these generated files should be added to .gitignore for good measure.

I'm happy to put some time and effort into this, but only if you think this is where we should go.

[twtg-martijn-bakker]

I went ahead and put in some effort. Please check out https://github.com/twtg-martijn-bakker/thingsboard-pe-docker-stack.git for a broad idea of how this should work.

I am working towards feature parity with this repository. Work is far from complete, but the current state should convey the general idea.

Main differences are:

• moved away from using scripts for managing the docker stack
• massive de-duplication
• focused on using docker-native mechanisms for load balancing, scaling etc.

[twtg-martijn-bakker]

closing this as it got stale, and we now built our own solution.