audit digest-mismatch errors

[dloy]

Audit has reported digest-mismatch for several files run on 2021-05-20.

Analysis of 3 files indicates concurrent ingests running on the same object.

2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-ingest.txt
2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-object-map.ttl
2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-erc.txt

20-May-2021 18:17:54.225 ... 192736 > POST http://uc3-mrtstore06x2-prd.cdlib.org:35121/update/9501/ark%3A%2F13030%2Fm5pk68rw

20-May-2021 18:18:04.791... 192736 < 200

---

20-May-2021 18:17:54.087 ... 192734 > POST http://uc3-mrtstore06x2-prd.cdlib.org:35121/update/9501/ark%3A%2F13030%2Fm5pk68rw

20-May-2021 18:18:04.580... 192734 < 200

Note the start and end times for these 2 runs overlap. There are high odds that the concurrent errors occurred because the manifest.xml was swapped out between the time of the initial inventory call and the completion of the 2nd ingest routine.

The full set of errors:

2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-ingest.txt
2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-object-map.ttl
2021-05-20 18:18:13 ark:/13030/m5pk68rw 4   system/mrt-erc.txt
2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-ingest.txt
2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-object-map.ttl
2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-erc.txt
2021-05-20 19:47:30 ark:/13030/m58q1t3r 8   system/mrt-ingest.txt
2021-05-20 19:47:30 ark:/13030/m58q1t3r 8   system/mrt-object-map.ttl
2021-05-20 19:47:30 ark:/13030/m58q1t3r 8   system/mrt-erc.txt
2021-05-20 19:47:31 ark:/13030/m58q1t3r 8   system/mrt-ingest.txt
2021-05-20 19:47:31 ark:/13030/m58q1t3r 8   system/mrt-object-map.ttl
2021-05-20 19:47:31 ark:/13030/m58q1t3r 8   system/mrt-erc.txt
2021-05-20 20:03:58 ark:/13030/m55f4kdb 11  system/mrt-ingest.txt
2021-05-20 20:03:58 ark:/13030/m55f4kdb 11  system/mrt-object-map.ttl
2021-05-20 20:03:58 ark:/13030/m55f4kdb 11  system/mrt-erc.txt
2021-05-20 20:09:38 ark:/13030/m5t49msc 3   system/mrt-ingest.txt
2021-05-20 20:09:38 ark:/13030/m5t49msc 3   system/mrt-object-map.ttl
2021-05-20 20:09:38 ark:/13030/m5t49msc 3   system/mrt-erc.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-ingest.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-object-map.ttl
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-erc.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-ingest.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-object-map.ttl
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-erc.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-ingest.txt
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-object-map.ttl
2021-05-20 21:41:19 ark:/13030/m5x9852v 2   system/mrt-erc.txt

[dloy]

Overlap ingests ark:/13030/m5tr1v78

2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-ingest.txt
2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-object-map.ttl
2021-05-20 18:54:18 ark:/13030/m5tr1v78 2   system/mrt-erc.txt

20-May-2021 18:53:16.677 ... 196623 > POST http://uc3-mrtstore05x2-prd.cdlib.org:35121/update/9501/ark%3A%2F13030%2Fm5tr1v78 ... 20-May-2021 18:53:24.000 INFO [http-nio-35121-exec-641] ... 196623 < 200

20-May-2021 18:53:18.750... 196631 > POST http://uc3-mrtstore05x2-prd.cdlib.org:35121/update/9501/ark%3A%2F13030%2Fm5tr1v78 ... 20-May-2021 18:53:21.101 INFO ... 196631 < 200

[dloy]

After group discussion. The fix for these problems is to delete these objects and do a resubmission.

List of arks:

ark:/13030/m5pk68rw
ark:/13030/m5tr1v78
ark:/13030/m58q1t3r
ark:/13030/m55f4kdb
ark:/13030/m5t49msc
ark:/13030/m5x9852v

[elopatin-uc3]

Thanks for the list @dloy

[dloy]

Note ark:/13030/m58q1t3r No overlap store but close inventories resulted in duplicate versions

start: 20-May-2021 19:45:10.576
stop:  20-May-2021 19:45:12.999

start: 20-May-2021 19:47:05.649
stop:  20-May-2021 19:47:12.064

Overlapping versions:

id  inv_object_id   ark number  note    created
4848120 3187328 ark:/13030/m58q1t3r 1   \r\n    2021-05-20 14:58:29
4848576 3187328 ark:/13030/m58q1t3r 2   \r\n    2021-05-20 15:54:16
4850292 3187328 ark:/13030/m58q1t3r 3   \r\n    2021-05-20 18:06:40
4851004 3187328 ark:/13030/m58q1t3r 4   \r\n    2021-05-20 18:39:03
4851195 3187328 ark:/13030/m58q1t3r 5   \r\n    2021-05-20 18:47:12
4852183 3187328 ark:/13030/m58q1t3r 6   \r\n    2021-05-20 19:27:58
4852530 3187328 ark:/13030/m58q1t3r 7   \r\n    2021-05-20 19:45:18
4852561 3187328 ark:/13030/m58q1t3r 8   \r\n    2021-05-20 19:47:30
4852565 3187328 ark:/13030/m58q1t3r 8   \r\n    2021-05-20 19:47:31
4853520 3187328 ark:/13030/m58q1t3r 9   \r\n    2021-05-20 20:38:57
4854487 3187328 ark:/13030/m58q1t3r 10  \r\n    2021-05-20 21:24:03
4854574 3187328 ark:/13030/m58q1t3r 11  \r\n    2021-05-20 21:27:37
4854715 3187328 ark:/13030/m58q1t3r 12  \r\n    2021-05-20 21:34:27
4856016 3187328 ark:/13030/m58q1t3r 13  \r\n    2021-05-20 22:35:10
4856072 3187328 ark:/13030/m58q1t3r 14  \r\n    2021-05-20 22:37:29
4856483 3187328 ark:/13030/m58q1t3r 15  \r\n    2021-05-20 22:55:57
4857481 3187328 ark:/13030/m58q1t3r 16  \r\n    2021-05-20 23:40:27
4858268 3187328 ark:/13030/m58q1t3r 17  \r\n    2021-05-21 00:12:47

---

Note ark:/13030/m55f4kdb Cross storage overlap

store06
start: 20-May-2021 20:03:07.806 
stop:  20-May-2021 20:03:30.574

store05
start: 20-May-2021 20:03:17.506
stop:  20-May-2021 20:03:24.015

---

Note: ark:/13030/m5t49msc Same store overlap - store06

start: 20-May-2021 20:09:02.964
stop:  20-May-2021 20:09:15.469 

start: 20-May-2021 20:09:03.207
stop:  20-May-2021 20:09:15.644

---

Note: ark:/13030/m5x9852v Same store overlap - store06

start: 20-May-2021 21:41:11.984
stop:  20-May-2021 21:41:14.321

start: 20-May-2021 21:41:13.439
stop:  20-May-2021 21:41:22.757 

[elopatin-uc3]

Complete list of images that will need to be re-ingested after deleting these ARKs. Note that I have contacted David T. at UCB to make sure he still has copies of these images in TIND.

m5pk68rw --
cubanc_e002801_01.tif
cubanc_e002801_02.tif
cubanc_e002801_03.tif
cubanc_e002801_04.tif
cubanc_e002801_05.tif
cubanc_e002801_06.tif
cubanc_e002801_07.tif
cubanc_e002801_08.tif
cubanc_e002801_09.tif
cubanc_e002801_10.tif
cubanc_e002801_11.tif
cubanc_e002801_12.tif
cubanc_e002801_13.tif
cubanc_e002801_15.tif
cubanc_e002801_16.tif
cubanc_e002801_17.tif

m5tr1v78 --
cubanc_e002660_02.tif 
cubanc_e002660_03.tif 
cubanc_e002660_04.tif 
cubanc_e002660_05.tif 
cubanc_e002660_06.tif 

m58q1t3r --
cubanc_e002334_01.tif
cubanc_e002334_02.tif
cubanc_e002334_03.tif
cubanc_e002334_04.tif
cubanc_e002334_06.tif
cubanc_e002334_07.tif
cubanc_e002334_08.tif
cubanc_e002334_09.tif
cubanc_e002334_10.tif
cubanc_e002334_11.tif
cubanc_e002334_12.tif
cubanc_e002334_13.tif
cubanc_e002334_14.tif
cubanc_e002334_15.tif
cubanc_e002334_16.tif
cubanc_e002334_17.tif
cubanc_e002334_18.tif

m55f4kdb --
cubanc_e003283_01.tif
cubanc_e003283_02.tif
cubanc_e003283_03.tif
cubanc_e003283_04.tif
cubanc_e003283_05.tif
cubanc_e003283_06.tif
cubanc_e003283_07.tif
cubanc_e003283_08.tif
cubanc_e003283_09.tif
cubanc_e003283_10.tif
cubanc_e003283_11.tif
cubanc_e003283_12.tif
cubanc_e003283_13.tif
cubanc_e003283_14.tif
cubanc_e003283_15.tif
cubanc_e003283_16.tif
cubanc_e003283_17.tif
cubanc_e003283_18.tif
cubanc_e003283_19.tif
cubanc_e003283_20.tif
cubanc_e003283_21.tif
cubanc_e003283_22.tif
cubanc_e003283_23.tif
cubanc_e003283_25.tif
cubanc_e003283_26.tif
cubanc_e003283_27.tif
cubanc_e003283_28.tif

m5t49msc --
cubanc_e003905_01.tif   
cubanc_e003905_02.tif   
cubanc_e003905_03.tif   
cubanc_e003905_05.tif   
cubanc_e003905_07.tif   
cubanc_e003905_08.tif   

m5x9852v --
cubanc_e003563_02.tif
cubanc_e003563_03.tif
cubanc_e003563_04.tif

[elopatin-uc3]

David T. has confirmed that he has not deleted any of these images from TIND on their side.

[elopatin-uc3]

Object deletes in production successful:

dpr...@uc3-...-prd:~/admin/deleteprod$ ./run.sh...
perl ./calldel.pl /apps/.../ucb_examiner_batch2.txt  /apps/.../...out/report-ucb_examiner_batch2-20210614-1042.txt
ark:/13030/m5pk68rw REPLIC=NONE STORE=OK INV=OK LOCAL=OK
ark:/13030/m5tr1v78 REPLIC=NONE STORE=OK INV=OK LOCAL=OK
ark:/13030/m58q1t3r REPLIC=NONE STORE=OK INV=OK LOCAL=OK
ark:/13030/m55f4kdb REPLIC=NONE STORE=OK INV=OK LOCAL=OK
ark:/13030/m5t49msc REPLIC=NONE STORE=OK INV=OK LOCAL=OK
ark:/13030/m5x9852v REPLIC=OK STORE=OK INV=OK LOCAL=OK

[elopatin-uc3]

Ingesting replacement, single-image objects via to_ingest_from_batch2.checkm bid-0e5a9ae2-c67a-4042-9cec-0735ea8f3317

74 objects/images

Job completed 10:51:52AM 6/14/2021