Get Stage Consistency Reports to a Valid State

[terrywbrady]

• [x] Delete content to match stage spreadsheet -- Mark
  • [x] Delete content older than 1 year for selected collections
  • [ ] Delete content failing size_mismatch, digest_mismatch
• [ ] Investigate remaining audit/unverified issues -- Terry
  • [ ] delete any objects that cannot be reconciled
  • [ ] rerun audit
• [ ] Review collection node configuration. -- Terry or Eric
  • [ ] All collections in the stage spreadsheet should have 3 nodes.
  • [ ] Empty collections that have not been deleted should have 3 nodes
  • [ ] Review "Collections with Non-standard Primary Nodes (Not SDSC or Dryad S3)" and code exceptions as needed
• [ ] Trigger replication for content not yet to 3 nodes - Terry
• [x] Investigate the 6 objects not in inv_nodes_inv_objects #1059 -- Terry

Brainstorm future stage needs

• [ ] Identify interesting test case Arks needed for retrieval testing - David
• [ ] Identify interesting test case Arks that would be good to re-ingest weekly as a good test of stage functionality - David

[terrywbrady]

MySQL [inv]> select distinct inv_collection_id, count(*)
    -> from inv_files f
    -> inner join inv_collections_inv_objects icio 
    ->   on icio.inv_object_id = f.inv_object_id 
    -> where billable_size > 0 
    -> and not exists (select 1 from inv_audits a where f.id=a.inv_file_id)
    -> group by inv_collection_id;

+-------------------+----------+
| inv_collection_id | count(*) |
+-------------------+----------+
|                24 |      117 |
|                29 |    57982 |
|                45 |       14 |
+-------------------+----------+

MySQL [inv]> select id, ark, erc_what from inv_objects o where not exists (select 1 from inv_nodes_inv_objects inio where inio.inv_object_id=o.id) and aggregate_role != 'MRT-collection';
+-------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| id    | ark                   | erc_what                                                                                                                                   |
+-------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 44885 | ark:/13030/aaa001     | President David Starr Jordan of Stanford speaking at Benjamin Ide Wheeler's inauguration, University of California at Berkeley. [negative] |
| 60487 | ark:/99999/testinvB   | Abandoned Apartment. Boston, Massachusetts, 1936                                                                                           |
| 64579 | ark:/99999/fk4ns20p10 | Test Node 9502 (minio)                                                                                                                     |
| 64589 | ark:/99999/fk4p56t73p | Test minio w/ large dataset                                                                                                                |
| 64595 | ark:/99999/fk4xs6z857 | random.img                                                                                                                                 |
| 64596 | ark:/99999/fk4698cq13 | random.img                                                                                                                                 |
| 64597 | ark:/99999/fk42j7nw72 | random.img                                                                                                                                 |
+-------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------+\