vpt HPC execution - Githubissues

Hello,

We've converted a Docker image to a Singularity image for execution in an HPC environment. After running for more than 3 hours and 120GB plus memory consumed, we are now being overloaded with the messages below.

2023-08-03 17:53:19,360 - ./task-20 - INFO - prepare_watershed_images 2023-08-03 17:54:43,495 - ./task-5 - INFO - watershed 2023-08-03 17:54:46,306 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.09 GiB -- Worker memory limit: 15.72 GiB 2023-08-03 17:54:51,999 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB 2023-08-03 17:54:52,099 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB 2023-08-03 17:54:52,199 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB

I terminated the process but would like to know if this is expected and what are the typical hardware resource requirements for this type of execution. Please let me know if you need further details and how can we further troubleshoot. Below are the parameters we are executing with:

singularity exec /home/USER/VPT_Run/vpt/vptlatest.sif vpt \ --verbose \ --processes 16 \ --log-file $OUTDIR'/run.log' \ --log-level 1 \ run-segmentation \ --segmentation-algorithm $OUTDIR'/watershed.json' \ --input-images $OUTDIR'/'$RUN'/mosaic(?P[A-Za-z]+)_z(?P[0-9]+).tif' \ --input-micron-to-mosaic $OUTDIR'/'$RUN'/micron_to_mosaic_pixel_transform.mat' \ --overwrite \ --output-path $OUTDIR'/'$RUN

Vizgen / vizgen-postprocessing

vpt HPC execution #18