haniffalab / webatlas-pipeline

A data pipeline built in Nextflow to process spatial and single-cell experiment data for visualisation in WebAtlas
MIT License
46 stars 10 forks source link

limitation of multithread writing to shared file system #67

Closed BioinfoTongLI closed 1 year ago

BioinfoTongLI commented 1 year ago

Causing host is down error when writing large number of files concurrently to a shared file system.

This is not an issue of this pipeline per se, but more a zarr issue I'd say. But good to keep in mind when converting large number of sample (60+ visium slides).