astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Unmanaged memory warnings in margin cache import #267

Open smcguire-cmu opened 4 months ago

smcguire-cmu commented 4 months ago

When running the margin cache import, there are a lot of repeated dask errors like this: distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information.

An example of this is in this notebook https://lsdb--166.org.readthedocs.build/en/166/notebooks/des-gaia.html