loicmathieu
commented
1 week ago Hi, Can you try with the new 0.17.5 release? It should have a fix for that.
Open nidomiro opened 1 week ago
loicmathieu
commented
1 week ago Hi, Can you try with the new 0.17.5 release? It should have a fix for that.
nidomiro
commented
1 week ago Unfortunately I can only confirm that the exception does not appear in the log anymore. The CPU and RAM usage is still the same. I also still need to hard-reset the system (an lxc-container running in Proxmox)
loicmathieu
commented
1 week ago Can you paste here your full flow YAML and the resources allocated to your container?
nidomiro
commented
1 week ago Sure. The value mySuperSecretKey is just a placeholder.
The flow yaml:
id: trigger-docker-server-autodeploy
namespace: hl443
description: Trigger autodeploy for all Docker servers
labels:
type: autodeploy
variables:
servers:
- fqn: docker01.hl443.de
user: root
- fqn: docker02.hl443.de
user: root
- fqn: docker03.hl443.de
user: root
- fqn: nextcloud.hl443.de
user: root
tasks:
- id: parallel
type: io.kestra.plugin.core.flow.EachParallel
value: "{{ vars.servers }}"
tasks:
- id: debugLog
type: io.kestra.plugin.core.log.Log
message:
- "{{ taskrun.value }}"
- id: trigger-autodeploy
type: io.kestra.plugin.fs.ssh.Command
host: "{{ json(taskrun.value)['fqn'] }}"
username: "{{ json(taskrun.value)['user'] }}"
authMethod: PUBLIC_KEY
privateKey: "{{ secret('SSH_ACCESS_KEY_' + json(taskrun.value)['fqn'] | replace({'.': '_'})) }}"
warningOnStdErr: false
commands:
- "source ~/.profile"
- "cd $HOMELAB_APPS_ROOT"
- "git pull"
- "./autodeploy.mts"
triggers:
- id: on-git-commit
type: io.kestra.plugin.core.trigger.Webhook
key: mySuperSecretKey
#key: "{{ secret('TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET') }}"
disabled: falseThe docker-compose:
version: "3.4"
services:
postgres:
image: postgres:16.3
restart: unless-stopped
volumes:
- ${HOMELAB_APPS_ROOT:?}/kestra/data/postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: ${KESTRA_POSTGRES_PASSWORD:?}
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 10s
retries: 10
networks:
- default
- proxynet
kestra:
image: kestra/kestra:v0.17.5-full
restart: unless-stopped
pull_policy: always
# Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
user: "root"
command: server standalone --worker-thread=128
volumes:
- ${HOMELAB_APPS_ROOT:?}/kestra/data/kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
env_file:
- ${HOMELAB_APPS_ROOT:?}/kestra/.env
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: ${KESTRA_POSTGRES_PASSWORD:?}
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # it must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://kestra.hl443.de/
labels:
traefik.enable: true
# Frontend
traefik.http.routers.kestra.rule: Host(`kestra.hl443.de`)
traefik.http.routers.kestra.entrypoints: websecure
traefik.http.routers.kestra.tls.certresolver: myresolver
traefik.http.services.kestra.loadbalancer.server.port: 8080
traefik.http.routers.kestra.service: kestra
traefik.http.routers.kestra-metrics.rule: Host(`kestra-metrics.hl443.de`)
traefik.http.routers.kestra-metrics.entrypoints: websecure
traefik.http.routers.kestra-metrics.tls.certresolver: myresolver
traefik.http.services.kestra-metrics.loadbalancer.server.port: 8081
traefik.http.routers.kestra-metrics.service: kestra-metrics
ports:
- 127.0.0.1:8080:8080
- 127.0.0.1:9080:8081
networks:
- default
- proxynet
depends_on:
postgres:
condition: service_started
networks:
default:
proxynet:
external: trueThe version of the .env file without values:
KESTRA_POSTGRES_PASSWORD=
SECRET_SSH_ACCESS_KEY_docker01_hl443_de=
SECRET_SSH_ACCESS_KEY_docker02_hl443_de=
SECRET_SSH_ACCESS_KEY_docker03_hl443_de=
SECRET_TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET=The LXC-Container config:
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:29:22 2024
OS/Arch: linux/amd64
Context: defaultServer: Docker Engine - Community Engine: Version: 26.1.4 API version: 1.45 (minimum version 1.24) Go version: go1.21.11 Git commit: de5c9cf Built: Wed Jun 5 11:29:22 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.33 GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957 runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0
I think I just found the cause. SECRET_TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET in the .env file was not base64 encoded but the original secret.
I encoded the value and now it works as expected.
Describe the issue
I created a webhook-trigger for my workflow. When the key is defined directly as literal, everything works. However If I use a secret, the system will go to >80% CPU and 100% RAM and needs to be hard-reset, when the webhook is triggerd.
I use a docker-compose file where the secrets are in a
.envfile and this file is referenced in the servicekestraviaenv_file:.the working trigger:
the "freezing" trigger:
here are the logs:
Environment