Open gtesoro opened 7 months ago
@gtesoro : Have you tried running the containers with the --init
flag (init: true
in docker-compose)?
According to the Docker reference, this will spawn a tini-like init process between dagster and the OS and does handle things like zombie processes.
Dagster version
1.6.8
What's the issue?
When using daemon-managed gRPC servers, the daemon seems to be hardcoded to refresh them every 60 seconds (RELOAD_WORKSPACE_INTERVAL). This seems to force the recreation of the gRPC servers for each code location (new pids) but leaves defunct python processes on the process. I took a look at the source code but the cause it's not clear to me since theoretically the _clear_old_processes() function from the GrpcServerRegistry should be taking care of polling and waiting to avoid zombies. The number of zombie processes seem to steadily increase which raised some flags on the IT department that manages the infrastructure where I'm deploying.
WORKAROUND:
What did you expect to happen?
No response
How to reproduce?
ps -eo pid, ppid, stat,cmd | grep Z
Deployment type
Docker Compose
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.