Open DanIredell-NOAA opened 8 months ago
Changed logic to use seq instead of brace expansion so can gzip any number of files as specified by fcstdays. old way: {06..96..6} # 96 is fcsthours but it cannot be a macro in a brace expansion new way: $(seq -w 06 06 $fcsthours)
Modified the gzip00 step to use pigz (parallel implementation of gzip) and it now takes 12 minutes to complete (5 minutes faster). This puts it under the 15 minute threshold. Will need to also investigate on using pigz for the gzip06 and gzip12 stepes, which take 11 minutes each.
When using pigz, jobs gzip06 and gzip12 improve from 11 minutes to 8 minutes each.
A couple of issues with the gzip step(s):