NOAA-EMC / RTOFS_GLO

0 stars 1 forks source link

GZIP step needs improvements #42

Open DanIredell-NOAA opened 8 months ago

DanIredell-NOAA commented 8 months ago

A couple of issues with the gzip step(s):

DanIredell-NOAA commented 8 months ago

Changed logic to use seq instead of brace expansion so can gzip any number of files as specified by fcstdays. old way: {06..96..6} # 96 is fcsthours but it cannot be a macro in a brace expansion new way: $(seq -w 06 06 $fcsthours)

Modified the gzip00 step to use pigz (parallel implementation of gzip) and it now takes 12 minutes to complete (5 minutes faster). This puts it under the 15 minute threshold. Will need to also investigate on using pigz for the gzip06 and gzip12 stepes, which take 11 minutes each.

DanIredell-NOAA commented 7 months ago

When using pigz, jobs gzip06 and gzip12 improve from 11 minutes to 8 minutes each.