Introduce docker cpu/memory limits to every container

[YuryHrytsuk]

Services:

• [x] adminer
• [x] admin-panels
• [x] deployment-agent
• [x] filestash
• [x] graylog
• [x] jaeger
• [x] mail (doesn't exist anymore)
• [x] maintenance-page (no need)
• [x] minio (no need since not used in production)
• [x] monitoring
• [x] pg-backup
• [x] portainer
• [x] redis-commander
• [x] registry
• [x] simcore
• [x] traefik

[YuryHrytsuk]

Overview of services as of Aug. 29th 2023

Admin panels

Service count: 1 Only memory limit is set

Admin

Service count: 1 Only memory limit & reservation is set

Depl. agent

Service count: 1 No limits / reservations

Filestash

Service count: 1 Only memory limit is set

Graylog

Service count: 3

mongodb: no limites / reservations elasticsearch: only memory limit graylog: all limits & reservations are set

Jaeger

Service count: 1 Only memory limit

Minio

Service count: 1 No limits / reservations

Monitoring

Service count: 14 Limits :

• memory: 14
• cpu: 0 Reservations:
• memory: 14
• cpu: 0

PG_BACKUP

Service Count: 1 No limits/reservations

Portainer

Service count: 2 No limits / reservations are set

Redis-commander

Service count: 1 Only memory limit & reservation are set

Registry

Service count: 1 No limits / reservations are set

Simcore

Service count: 18

Traefik

Service count: 2

Traefik:

• memory & cpu limits & reservations are set

whoami:

• no limits / reservations are set

[mrnicegyu11]

We do not work on this before we an observe that it in production and be sure it doesn't break anything

[YuryHrytsuk]

How to evaluate container CPU usage

sum(rate(container_cpu_usage_seconds_total{container_label_com_docker_swarm_service_name="<service_name>"}[1m])) by (name)

• rate - calculate the per-second average rate of how a value is increasing over a period of time
• metric container_cpu_usage_seconds_total is ever increasing line as name would suggest. With rate function we are able to detect (in milliseconds) the CPU usage per second by a container
  • rate(container_cpu_usage_seconds_total[<time frame>]) metric finally presents CPU Usage. I don't fully understand why it does but I found a few proofs on the internet
  • this is the best article I found that explains how to convert this metric to CPU Usage. It is also very helpful to understand the whole query in general
• without sum we will get the usage per every available CPU, e.g. 1ms of cpu1, 2ms of cpu2, and so on, i.e. we have different records for the same container with the only difference in the cpu used. By summing them up, we get total CPU usage across all the CPUs
• by name is a grouping by container name. It plays nicely since it automatically summs the usage per every CPU core and output the total CPU usage per container

How to evaluate container memory

container_memory_usage_bytes{name="<container_name>"}

[YuryHrytsuk]

Evaluating CPU Usage logic should be improved. It doesn't return meaningful results

[mrnicegyu11]

Tracks duplicate https://github.com/ITISFoundation/osparc-ops-environments/issues/231