Add support for Proxmox (either plugin or the ability to run kestra within proxmox)

[anna-geller]

Feature description

details TBD, waiting for input from users

[eternity336]

Proxmox uses lxc containers instead of docker. Requesting integration into Proxmox to have Kestra as part of the Turnkey products that you can download as containers. This way we can get the same Docker experience on Proxmox with full plugin at install. Proxmox is also a VM and container environment similar to ESXi, and it would be nice to see plugin support for managing the container/vm environment.

[anna-geller]

extra info: In the meantime, this may be helpful https://github.com/Telmate/terraform-provider-proxmox

[fuzzy]

FWIW I've had several issues with the telmate provider that I've not been able to track down as of yet, I've had good success with this provider however: https://github.com/bpg/terraform-provider-proxmox YMMV of course, but I thought it would be prudent to note other options.

[bwalocha]

I am not able to get it working:

kestra  | 2024-06-11 17:23:18,750 INFO  worker_0     flow.hello_world.first_task [namespace: tutorial] [flow: hello_world] [task: first_task] [execution: 70RNNvz7jYbTZt1hstEpfg] [taskrun: 4nV1bOQRgUf5NSSBFm2NZB] Type Return started
kestra  | 2024-06-11 17:23:18,854 INFO  worker_0     flow.hello_world.first_task [namespace: tutorial] [flow: hello_world] [task: first_task] [execution: 70RNNvz7jYbTZt1hstEpfg] [taskrun: 4nV1bOQRgUf5NSSBFm2NZB] Type Return with state SUCCESS completed in 00:00:01.120
kestra  | 2024-06-11 17:23:19,917 INFO  worker_1     flow.hello_world.second_task [namespace: tutorial] [flow: hello_world] [task: second_task] [execution: 70RNNvz7jYbTZt1hstEpfg] [taskrun: 6R1SGd6exGnwrScL0mjPtA] Type Commands started
kestra  | 2024-06-11 17:23:24,745 ERROR WorkerThread f.h.7.6R1SGd6exGnwrScL0mjPtA Status 500: {"message":"failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply caps: operation not permitted: unknown"}
kestra  |
kestra  | com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply caps: operation not permitted: unknown"}
kestra  |
kestra  |       at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
kestra  |       at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:102)
kestra  |       at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:31)
kestra  |       at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:13)
kestra  |       at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
kestra  |       at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
kestra  |       at com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:42)
kestra  |       at io.kestra.plugin.scripts.runner.docker.Docker.run(Docker.java:257)
kestra  |       at io.kestra.plugin.scripts.exec.scripts.runners.CommandsWrapper.run(CommandsWrapper.java:159)
kestra  |       at io.kestra.plugin.scripts.shell.Commands.run(Commands.java:134)
kestra  |       at io.kestra.plugin.scripts.shell.Commands.run(Commands.java:18)
kestra  |       at io.kestra.core.runners.WorkerTaskThread.doRun(WorkerTaskThread.java:77)
kestra  |       at io.kestra.core.runners.AbstractWorkerThread.run(AbstractWorkerThread.java:56)

My docker compose:

secrets:
  app_config:
    file: ./secrets/kestra/app_config.secret

services:
  kestra:
    # https://github.com/kestra-io/kestra
    image: kestra/kestra:$TAG_KESTRA
    container_name: kestra
    privileged: true
    security_opt:
      # - no-new-privileges:true
      - seccomp:unconfined
      - apparmor:unconfined
    cap_add:
      - ALL
    user: "root"
    restart: "no"
    command: server standalone --config /run/secrets/app_config --worker-thread=128
    depends_on:
      traefik:
        condition: service_started
      postgres:
        condition: service_started
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/kestra/storage:/app/storage:rw
      - ./conf/kestra/config:/app/config:ro
      - ./logs/kestra/tmp:/tmp/kestra-wd:rw
    secrets:
      - app_config
    labels:
      traefik.enable: true
      # Https
      traefik.http.routers.kestra.entrypoints: websecure
      traefik.http.routers.kestra.rule: Host(`kestra.$DOMAIN`)
      traefik.http.routers.kestra.tls.certresolver: certResolver
      # Middlewares:
      traefik.http.routers.kestra.middlewares: chain-oauth@file
      # Services:
      traefik.http.services.kestra.loadbalancer.server.port: 8080

Kestra 0.17.1 Proxmox: 8.2.2 + LXC with Privileged mode

Can anybody help? I tried all combinations with and without security_opt, cap_add - the same result.

[nidomiro]

I run kestra as a docker-container in an unprivileged lxc container with nesting enabled and had zero problems in that regard so far. I used this as a template: https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml and basically only changed env-vars and some secrets

[anna-geller]

@nidomiro are you willing to share your Docker Compose file here (redacted) or contribute to our docs to show how to use Proxmox? Thanks so much in advance!

[nidomiro]

I already posted in in another issue, you can find it here: https://github.com/kestra-io/kestra/issues/4095#issuecomment-2187215054

[bwalocha]

In my case I have privileged container:

with Nesting turned on. The compose file:

secrets:
  app_config:
    file: ./secrets/kestra/app_config.secret

services:
  kestra:
    # https://github.com/kestra-io/kestra
    image: kestra/kestra:$TAG_KESTRA
    container_name: kestra
    privileged: true
    user: "root"
    restart: "no"
    command: server standalone --config /run/secrets/app_config --worker-thread=128
    depends_on:
      traefik:
        condition: service_started
      postgres:
        condition: service_started
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/kestra/storage:/app/storage:rw
      - ./logs/kestra/tmp:/tmp/kestra-wd:rw
    secrets:
      - app_config
    # environment:
    #   KESTRA_CONFIGURATION: |
        # datasources:
        #   postgres: # mysql: # postgres:
        #     url: jdbc:postgresql://postgres:5432/kestra # jdbc:mysql://mariadb:3306/kestra?useUnicode=true&characterEncoding=UTF-8&serverTimezone=${TZ} # jdbc:postgresql://postgres:5432/kestra
        #     driverClassName: org.postgresql.Driver # com.mysql.cj.jdbc.Driver # org.postgresql.Driver
        #     username: ${PQ_USER} # root
        #     password: ${PQ_PASS} # ${MARIADB_ROOT_PASSWORD}
        # kestra:
        #   server:
        #     basic-auth:
        #       enabled: false
        #       username: ${ADMIN_EMAIL} # it must be a valid email address
        #       password: ${KESTRA_PASSWORD}
        #   repository:
        #     type: postgres # mysql # postgres
        #   storage:
        #     type: local
        #     local:
        #       base-path: "/app/storage"
        #   queue:
        #     type: postgres # mysql # postgres
        #   tasks:
        #     tmp-dir:
        #       path: /tmp/kestra-wd/tmp
        #   url: http://localhost:8080/
    labels:
      traefik.enable: true
      # Https
      traefik.http.routers.kestra.entrypoints: websecure
      traefik.http.routers.kestra.rule: Host(`kestra.$DOMAIN`)
      traefik.http.routers.kestra.tls.certresolver: certResolver
      # Middlewares:
      traefik.http.routers.kestra.middlewares: chain-oauth@file
      # Services:
      traefik.http.services.kestra.loadbalancer.server.port: 8080

and when I try to run hello word example I get the error

failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply caps: operation not permitted: unknown