search

Illumina / pyflow

A lightweight parallel task engine
http://Illumina.github.io/pyflow/
146 stars 44 forks source link

force a sync so reads from shared FS don't fail #25

Closed koebbe closed 4 years ago

koebbe commented 4 years ago

Some distributed file systems can be configured to not sync on file close. This can cause reads of the pickled data fail. This change forces a sync (flush) of the data to prevent potential future read failures.

koebbe commented 4 years ago

This prevents the error mentioned in Illumina/strelka#69

ctsa commented 4 years ago

Makes sense, thanks for finding this.