3.3.10. Logs

[Pcolar]

3.3.10.1. Operational

• Logs should not be archived
• How will space management be handled?
• Who is responsible for management?

[acarrasco-libnova]

The logs are stored in the network shared file system, the responsible of the management of that storage is the USC.

[Pcolar]

Note: needs to be addressed in the USC-DPN SLA

[wmccown]

This is what I see on the Network file system under the "prod/logs":

root@dr-nl400-10 /ifs/home/dpn/prod/logs]# ls -l total 25405886 -rwxrwx--- + 1 1003772 1000831 6953602217 Jul 13 23:00 LS_Archiver.log -rwxrwx--- + 1 1003772 1000831 23349775 Jul 13 23:00 LS_Auditor.log -rwxrwx--- + 1 1003772 1000831 4318214403 Jul 13 23:01 LS_Characterizer.log -rwxrwx--- + 1 1003772 1000831 1686952289 Jul 13 23:00 LS_Checker.log -rwxrwx--- + 1 1003772 1000831 6494499250 Jul 13 23:00 LS_Verifier.log drwxrwx--- + 2 1003775 1000831 273663 Apr 30 08:56 audit drwxrwx--- + 2 1003775 1000831 1119 Apr 25 02:43 deposit

The "audit" folder contains some 10,547 log files that are just sequentially numbered. The numbering appears to have been rolled on April 30 of this year and the sequence started over at 1. The newest is dated June 2. The deposit folder also has a bunch of sequentially numbered files, the newest is May 1.

Total space consumed at the moment is 31GiB.

[acarrasco-libnova]

Well, about those logs I must say that the libsafe modules on the USC instance are outdated and right now the logs look kinda different in our demo instance (exanode-demo.libnova.com):

C:\libsafe\logs>dir /a
 Volume in drive C has no label.
 Volume Serial Number is 927D-C8FA

 Directory of C:\libsafe\logs

07/17/2018  06:08 PM    <DIR>          .
07/17/2018  06:08 PM    <DIR>          ..
06/27/2018  11:25 PM               422 20180626_LS_Archiver.log
06/27/2018  11:24 PM               452 20180626_LS_Characterizer.log
06/27/2018  11:24 PM               140 20180626_LS_Checker.log
06/29/2018  12:30 PM               143 20180627_LS_Archiver.log
06/29/2018  12:29 PM               153 20180627_LS_Characterizer.log
06/29/2018  12:29 PM               140 20180627_LS_Checker.log
06/26/2018  02:19 PM               819 20180628_LS_Archiver.log
06/26/2018  02:18 PM             2,534 20180628_LS_Characterizer.log
06/26/2018  02:51 PM             7,044 20180628_LS_Checker.log
06/29/2018  12:30 PM                 0 20180629_LS_Archiver.log
06/29/2018  12:30 PM                 0 20180629_LS_Characterizer.log
06/29/2018  12:29 PM                 0 20180629_LS_Checker.log
07/11/2018  11:06 AM               819 20180711_LS_Archiver.log
07/11/2018  04:57 PM            16,076 20180711_LS_Auditor.log
07/11/2018  11:05 AM             6,369 20180711_LS_Characterizer.log
07/11/2018  11:04 AM               808 20180711_LS_Checker.log
07/13/2018  04:57 PM               680 20180713_LS_Archiver.log
07/13/2018  04:57 PM             3,157 20180713_LS_Characterizer.log
07/17/2018  02:19 PM               809 20180713_LS_Checker.log
07/17/2018  06:08 PM                 0 20180717_LS_Archiver.log
07/17/2018  04:18 PM             1,143 20180717_LS_Auditor.log
07/17/2018  06:08 PM                 0 20180717_LS_Characterizer.log
07/17/2018  06:08 PM                 0 20180717_LS_Checker.log
07/17/2018  04:14 PM             1,466 20180717_LS_Verifier.log
07/11/2018  04:57 PM    <DIR>          audit
06/29/2018  12:29 PM    <DIR>          deposit
              24 File(s)         43,174 bytes

The file name scheme for the log files now is YYYYMMDD_{module}.log. I have updated the Technical document to reflect that, in point 4.4.3. Logs cleaning.

Inside the audit folder there should be the logs of the audit processes, and the deposit folder will store the specific deposit log files (the ingestion process log, which can be downloaded from the Web Interface).

[Pcolar]

A draft of the SLA is at USC for review. DPN assumes that this task will be managed by USC, under terms included in the SLA.

[wmccown]

Dave: Just what do you want the SLA to say on the subject? Is your first bullet point above still valid:

Logs should not be archived

If so then the Isilon storage is just fine for the purpose. That data can be replicated to Clemson if you like for disaster recovery purposes, and with a reasonable log rotating policy should not become a significant amount of data.

[Pcolar]

Will: That is a direct quote from the Technical document 3.3.10.1. Operational The information in the logs belonging to this class is relevant as Digital Preservation Information, that should always be part of the Exanode system. Logs should not be archived periodically here.

Content will be retained on the Isilon and replicated to Clemson.

[Pcolar]

This issue will be negotiated in the SLA.