archivematica / Issues

Issues repository for the Archivematica project
GNU Affero General Public License v3.0
16 stars 1 forks source link

Problem: Reduce the verbosity of the email fail report script #101

Open ross-spencer opened 6 years ago

ross-spencer commented 6 years ago

Please describe the problem you'd like to be solved.

Currently, the output in the logs for the email fail report is quite verbose as it contains the entire HTML formatted message:

archivematica-mcp-client_1       | *** RUNNING TASK: emailfailreport_v0.0
archivematica-mcp-client_1       | archivematicaClient.py: INFO      2018-08-16 21:59:00,260  archivematica.mcp.client.emailFailReport:get_content_for:146:  Generating report content (unit_type=Transfer, unit_uuid=2bc98520-6814-47be-8c6e-89d7a530f429, html=True)
archivematica-mcp-client_1       | archivematicaClient.py: INFO      2018-08-16 21:59:00,290  archivematica.mcp.client.emailFailReport:send_email:53:  Sending email...
archivematica-mcp-client_1       | Content-Type: multipart/alternative;
archivematica-mcp-client_1       |  boundary="===============6610532838109694571=="
archivematica-mcp-client_1       | MIME-Version: 1.0
archivematica-mcp-client_1       | Subject: Archivematica Fail Report for Transfer:
archivematica-mcp-client_1       |  2-2bc98520-6814-47be-8c6e-89d7a530f429
archivematica-mcp-client_1       | From: webmaster@example.com
archivematica-mcp-client_1       | To: test@test.com
archivematica-mcp-client_1       | Date: Thu, 16 Aug 2018 21:59:00 -0000
archivematica-mcp-client_1       | Message-ID: <20180816215900.8.85864@018130aeee3b>
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | --===============6610532838109694571==
archivematica-mcp-client_1       | MIME-Version: 1.0
archivematica-mcp-client_1       | Content-Type: text/plain; charset="utf-8"
archivematica-mcp-client_1       | Content-Transfer-Encoding: 7bit
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | Please see the attached HTML document
archivematica-mcp-client_1       | --===============6610532838109694571==
archivematica-mcp-client_1       | MIME-Version: 1.0
archivematica-mcp-client_1       | Content-Type: text/html; charset="utf-8"
archivematica-mcp-client_1       | Content-Transfer-Encoding: 7bit
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | <HTML>
archivematica-mcp-client_1       |   <BODY>
archivematica-mcp-client_1       |     <table cellpadding="4" style="border: 1px solid #000000; border-collapse: collapse;" border="1">
archivematica-mcp-client_1       |       <tr><th>Unit type</th>
archivematica-mcp-client_1       |   <th>Total time processing</th>
archivematica-mcp-client_1       |   <th>Total file size</th>
archivematica-mcp-client_1       |   <th>Number of files</th>
archivematica-mcp-client_1       |   <th>Average file size KB</th>
archivematica-mcp-client_1       |   <th>Average file size MB</th>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr><td>Transfer</td>
archivematica-mcp-client_1       |   <td>00:00:06</td>
archivematica-mcp-client_1       |   <td>1</td>
archivematica-mcp-client_1       |   <td>3039</td>
archivematica-mcp-client_1       |   <td>3.03900000</td>
archivematica-mcp-client_1       |   <td>0.00303900</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |     </table>
archivematica-mcp-client_1       |     <p/>
archivematica-mcp-client_1       |     <table cellpadding="4" style="border: 1px solid #000000; border-collapse: collapse;" border="1">
archivematica-mcp-client_1       |       <tr><th>Type</th>
archivematica-mcp-client_1       |   <th>Status</th>
archivematica-mcp-client_1       |   <th>Started</th>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#f2dede"><td>Parse Dataverse METS XML</td>
archivematica-mcp-client_1       |   <td>4</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:59.686135+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Determine Parse Dataverse METS XML</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:59.650147+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Validate formats</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:58.639264+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Load labels from metadata/file_labels.csv</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:58.236623+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Characterize and extract metadata</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:57.027945+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Load rights</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:56.625741+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Store file modification dates</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:56.199165+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Add processed structMap to METS.xml document</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:55.847435+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Extract packages?</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:55.815284+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Determine if transfer contains packages</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:55.541353+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to extract packages</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:54.602099+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Identify file format</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       | archivematicaClient.py: INFO      2018-08-16 21:59:00,295  archivematica.mcp.client.emailFailReport:get_content_for:146:  Generating report content (unit_type=Transfer, unit_uuid=2bc98520-6814-47be-8c6e-89d7a530f429, html=False)
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:53.553499+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Determine which files to identify</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:53.536227+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Select file format identification command</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:53.501167+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to select file ID tool</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:53.175297+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Sanitize Transfer name</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:52.790643+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Sanitize object's file and directory names</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:52.382670+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Save directory tree</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:52.003681+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Generate transfer structure report</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:51.977345+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to processing directory</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:51.660110+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to generate transfer tree</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:51.374118+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Scan for viruses</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:49.970344+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to processing directory</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:49.658621+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Workflow decision - send transfer to quarantine</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:49.618006+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Find type to process as</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:49.580545+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Move to workFlowDecisions-quarantineSIP directory</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:48.781093+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Designate to process as a standard transfer</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:48.746572+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Generate METS.xml document</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:48.344286+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Verify metadata directory checksums</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:47.943569+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Process transfer JSON metadata</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:47.539983+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Assign checksums and file sizes to objects</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:47.134800+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Assign file UUIDs to objects</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:46.723830+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Assign UUIDs to directories</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:46.368192+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Assign UUIDs to directories?</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:46.320417+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Include default Transfer processingMCP.xml</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:45.914635+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Rename with transfer UUID</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:45.500398+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Verify mets_structmap.xml compliance</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:45.104346+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Verify transfer compliance</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:44.696161+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Attempt restructure for compliance</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:44.275706+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Convert Dataverse Structure</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.914256+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Determine Dataverse conversion</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.886104+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Remove unneeded files</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.483026+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Remove hidden files and directories</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.162602+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Set Parse Dataverse Mets</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.128534+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Set Convert Dataverse Structure</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:43.075411+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |       <tr bgcolor="#dff0d8"><td>Set transfer type: Dataverse</td>
archivematica-mcp-client_1       |   <td>2</td>
archivematica-mcp-client_1       |   <td>2018-08-16 21:58:42.728398+00:00</td>
archivematica-mcp-client_1       |  </tr>
archivematica-mcp-client_1       |     </table>
archivematica-mcp-client_1       |   </BODY>
archivematica-mcp-client_1       | </HTML>
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | --===============6610532838109694571==--
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | -------------------------------------------------------------------------------
archivematica-mcp-client_1       | archivematicaClient.py: INFO      2018-08-16 21:59:00,306  archivematica.mcp.client.emailFailReport:store_report:180:  Storing report in database (unit_type=Transfer, unit_uuid=2bc98520-6814-47be-8c6e-89d7a530f429)
archivematica-mcp-client_1       | archivematicaClient.py: INFO      2018-08-16 21:59:00,332  archivematica.mcp.client:write_task_results_callback:200:  
archivematica-mcp-client_1       | 
archivematica-mcp-client_1       | *** Completed job: #<emailfailreport_v0.0; exit=0; code=success uuid=6aac5799-9338-4026-a929-f438dbdb02ed

Describe the solution you'd like to see implemented.

I'd like to see a neater summary generated that provides developers with the logging they need for this particular script without outputting the entire content of this mail.

Describe alternatives you've considered.

Changing the logging level for this output from info to debug should (I think) also help devs see the information they need without removing this verbosity entirely. That might be worth a try.

Additional context

I am currently working in qa/0.x branch.

sevein commented 6 years ago

Changing the logging level for this output from info to debug should (I think) also help devs see the information they need without removing this verbosity entirely. That might be worth a try.

Our default email backend is django.core.mail.backends.console.EmailBackend which is not using the logger but it's writing to sys.stdout. A good solution could be to use the file backend (https://docs.djangoproject.com/en/dev/topics/email/#file-backend) but I'm not sure what'd be a good place to put the file, maybe in the tmp inside the AM shared directory?

You can change the backend plus other settings via environment strings. They're listed in our install/ READMEs. We don't have an overview of email config in our docs yet but there are some notes here: https://github.com/artefactual/archivematica/pull/813.