gRPC servers spawned by the dagster-daemon seem to create zombie processes

[gtesoro]

Dagster version

1.6.8

What's the issue?

When using daemon-managed gRPC servers, the daemon seems to be hardcoded to refresh them every 60 seconds (RELOAD_WORKSPACE_INTERVAL). This seems to force the recreation of the gRPC servers for each code location (new pids) but leaves defunct python processes on the process. I took a look at the source code but the cause it's not clear to me since theoretically the _clear_old_processes() function from the GrpcServerRegistry should be taking care of polling and waiting to avoid zombies. The number of zombie processes seem to steadily increase which raised some flags on the IT department that manages the infrastructure where I'm deploying.

WORKAROUND:

• Spawn manually a gRPC process per code location and configure the workspace.yml to use them instead of python module/files

What did you expect to happen?

No response

How to reproduce?

• Deploy dagster-demon with managed gRPC servers and 1+ code location that contains a direct python module/file
• Watch for zombie processes ps -eo pid, ppid, stat,cmd | grep Z

Deployment type

Docker Compose

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[fbardos]

@gtesoro : Have you tried running the containers with the --init flag (init: true in docker-compose)?

According to the Docker reference, this will spawn a tini-like init process between dagster and the OS and does handle things like zombie processes.