search

AllenInstitute / datacube

Other
0 stars 1 forks source link

automated dev deployment accumulating open file handles to deleted data files #103

Closed chrisbarber closed 5 years ago

chrisbarber commented 5 years ago

du /local2 was showing 13G used, while df /local2 was showing 17G used. The discrepancy was due to: https://askubuntu.com/a/390301

Running lsof | grep '(deleted)' on devdatacube as root produced:

python    27280              mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27298        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27299        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27300        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27301        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27302        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27280 27303        mongrel    8rR     REG              252,1     255345694     282529 /local2/datacube_data/bob_data/cell_specimens.nc (deleted)
python    27283              mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283              mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283              mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27306        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27306        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27306        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27307        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27307        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27307        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27308        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27308        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27308        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27343        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27343        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27343        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27344        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27344        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27344        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27345        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27345        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27345        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27346        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27346        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27346        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27347        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27347        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27347        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27349        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27349        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27349        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27350        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27350        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27350        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27374        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27374        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27374        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27375        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27375        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27375        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27376        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27376        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27376        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27377        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27377        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27377        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)
python    27283 27378        mongrel    8r      REG              252,1 1099511627776     282538 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/data.mdb (deleted)
python    27283 27378        mongrel    9ur     REG              252,1          8192     282537 /local2/datacube_data/conn_data/connectivity.zarr.lmdb/lock.mdb (deleted)
python    27283 27378        mongrel   12rR     REG              252,1     693988542     282546 /local2/datacube_data/mouse_ccf_data/mouse_ccf.nc (deleted)

A systemctl stop crossbar.service was enough to clear these. It's unclear why the automated deployment wouldn't do the same, since it should be doing a stop/start or restart just the same.

chrisbarber commented 5 years ago

Not sure why but it looks like this is no longer an issue.