Hitting OSError: [Errno 24] Too many open files

[natekupp]

Places this has surfaced:

• @catherinewu can reproduce reliably by running pytest python_modules/libraries/dagster-dask/dagster_dask_tests/test_execute.py (but I cannot)
• @helloworld ran into it while clicking around dagit
• @yuhan ran into it while running dagster core tests
• @alangenfeld ran into it with graphql tests

It's not clear that these are all related.

Here's one way to reproduce:

ulimit 256
pytest -s python_modules/dagster-graphql/dagster_graphql_tests/graphql/

To observe file descriptors leaking, run this in one terminal:

pytest -s python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_config_types.py

and repeatedly run this in another terminal:

lsof -p $(ps -A | grep pytest | grep -v grep | awk '{print $1}') > $(date +"%s")

Observe that the number of open file descriptors given by lsof increases throughout the test, with about one new temporary directory sticking around per test.

[natekupp]

Seems best lead is generating lots of IPC calls

[natekupp]

https://mg.pov.lt/objgraph/ maybe useful

[alangenfeld]

https://dagster.phacility.com/P59

[sryza]

Observations:

• objgraph indicates no more than one TemporaryDirectory objects on the heap
• The main open fd's for test_config_types are temporary files/directories, while the main open fd's for the dask tests are pipes, implying that something different is going on between them.
• Only one or two file descriptors are being leaked per test_config_types test.
• The fd's left open aren't the temporary directories created in the InstanceManagers of graphql_context_test_suite.

[sryza]

Found it: https://dagster.phacility.com/D3472

[schrockn]

https://dagster.phacility.com/D3472