search

niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"
BSD 3-Clause "New" or "Revised" License
5 stars 1 forks source link

bioformats2raw still taking v long time on HPC #391

Open philipmac opened 9 months ago

philipmac commented 9 months ago

Trying to run: /gs1/apps/user/spack-0.16.0/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/bioformats2raw-0.7.0-7kt7dff7f7fxmdjdk57u6xjuzmsxqodn/bin/bioformats2raw --max_workers=3 --overwrite --compression blosc --compression-properties cname=zstd --compression-properties clevel=5 --compression-properties shuffle=1 --tile_width 512 --tile_height 512 --downsample-type AREA /mnt/ai-fas12/RMLSOHedwigQA/Projects/RTB/darrellh/nguyenm8-2023-1127-test-hn-2023-11-27/Protein-2023-1127-test-hn-2023-11-27-test-hn-2023-11-27/HQ_Images/IA_M4_S2.czi /gs1/Scratch/hedwig_dev_scratch/tmpkzw_aupr/IA_M4_S2.zarr 07:51:10 PM -> 09:18:24 PM

annshress commented 9 months ago

Possible reason - uncertainty in the the underlying hardware (different node capabilities, how busy the node was). Following two runs had the same params but started at different times shows different run times:

image