Two 2D pipeline runs failed in Prod - 12/19/2023

niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"

BSD 3-Clause "New" or "Revised" License

5 stars 1 forks source link

Two 2D pipeline runs failed in Prod - 12/19/2023 #413

Closed NetaFG closed 8 months ago

NetaFG commented 8 months ago

This issue was reported by Bryan H,

The pipeline runs: https://prefect1.hedwig-workflow-api.niaidprod.net/default/flow-run/8c095892-ac26-4099-9779-377022309d76?schematic https://prefect1.hedwig-workflow-api.niaidprod.net/default/flow-run/8c095892-ac26-4099-9779-377022309d76

It seems that after these failed runs, the user reran this pipeline, and the run completed successfully. Please investigate.

philipmac commented 8 months ago

Disk quota exceeded: '/gs1/Scratch'

Filesystem    Size  Used Avail Use% Mounted on
/gs1              3.6P  2.6P  1.1P  72% /gs1

if the filesystem is claiming its quota is exceeded one time in the past three months during which two runs were submitted. This issue was then resolved within HPC, and subsequent runs work. There is nothing we can do about this, although I can mention it to HPC admins.