search

kestra-io / kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
https://kestra.io
Apache License 2.0
7.45k stars 445 forks source link

Use a volume instead of a bind in the Docker task runner #3857

Closed loicmathieu closed 1 month ago

loicmathieu commented 2 months ago

Issue description

To work with files, the Docker task runner binds the task working directory inside the container. This causes some issues and complexities (for ex for Windows to Linux).

One of biggest issue is that if the container engine is launched with the root user but Kestra with another user, files created inside the bonded path will not be deleteable by Kestra leading to an error in the log and the file polluting the filesystem.

To fix that, we can create a volume and copy the working directory files in it before launching the container, then retreiving the file before deleting the container.

anna-geller commented 1 month ago

is this needed for task runner GA? if not, let's move to 19 or 20

loicmathieu commented 1 month ago

This would potentially fix a bunch of issues for Windows or non-standard Linux installation so it's a nice to have

anna-geller commented 1 month ago

this will likely fix this issue https://github.com/kestra-io/kestra/issues/4348