search

ACCESS-NRI / accessdev-Trac-archive

Archive accessdev Trac contents as issues
Apache License 2.0
0 stars 0 forks source link

Log file archiving #348

Open penguian opened 6 years ago

penguian commented 6 years ago

| by mrd599@nci.org.au

Coupled model log files can be very large.

Rose has the ability to automatically tar and gzip.

Issue migrated from trac:348 at 2024-01-31 18:32:45 +1100

penguian commented 6 years ago

@martin.dix@anu.edu.au commented

In Dave's u-ar527, log file from a 6 month run is 67 MB!

% ls -l 00200101T0000Z/coupled/01     
total 68388
-rwxr-xr-x 1 dhb599 p66     8210 Nov  7 01:58 job
-rw-r----- 1 dhb599 p66   889959 Nov  7 05:19 job.err
-rw-r----- 1 dhb599 p66 69114325 Nov  7 05:22 job.out
-rw-r--r-- 1 dhb599 p66      237 Nov  7 05:22 job.status

Splitting the job.out file into parts from the separate models shows that only trivial amounts are from cylc and CICE and that there's about 50 MB from MOM and 19 from the UM.

penguian commented 6 years ago

@martin.dix@anu.edu.au commented

Using the archive-logs-at option in rose-prune creates a .tar.gz file on the server (accessdev) and deletes the log files on raijin.

For this to be useful we'd need to greatly increase the size limit on log files that are copied back to accessdev.