Store Object "reset" - Githubissues

dloy commented 7 months ago

Object reset would be a mechanism to allow a user to remove 1 or more current versions of an object.

Features:

would allow 1 or more versions from the current version to be kept - Example keep=3 (if 10 versions this would save versions 8, 9, and 10
delete of an object is not allowed with reset.
keep= would be the maximum completion numbers
minimize remaining object size

Questions about this process:

Is an add version allowed with a reset - the added version would be treated as current?
- fits model of published version
should local id be allowed for specifying a reset or ark only?
with a reset is version count for file restarted as one (1). If key format changes then yes. If not possible collisions on key values.
should key be changed to ark ! sha256 ! length (?) - allows identification of file without fileid - not version centric.
is Oops required?
Not apparent in scan where an unmatched item came from

Changes required

Would be first delete feature provided by users.

Manifest

(suggest) versioning always reset to 1 in manifest
new flag indicating a reset

Ingest

if an "addreset" is allowed then ingest would treat as normal add - call to storage would require a new property on add request for reset in storage

Storage

would require digest matching for stored key across object
keys for all remaining versions would need to be reset in manifest. List of unused content would be deleted. No rewrite on key
deleted content would include duplicated content because of new key
add manifest reset flag

Inventory

reset would remove object and then process incoming save manifest.xml with all content. With versionID resetting would work as now

Replication

changed to be sensitive to reset flag in manifest - do delete for object on secondary
rebuild object from manifest - would work as now

Scan

to be worked out

terrywbrady commented 7 months ago

I started some brainstorming about use cases here. https://github.com/CDLUC3/mrt-doc/blob/main/design/object-reset/README.md

dloy commented 7 months ago

All non ?changeToken file paths in prod using

SELECT f.pathname FROM inv_files f WHERE f.id IN ( SELECT f.id FROM inv_files f WHERE f.billable_size > 0 AND f.pathname LIKE '%?%=%' AND NOT f.pathname LIKE '%changeToken%' )

producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=101
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=333
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/wp-includes/js/comment-reply.js?ver=20090102
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/player.vimeo.com/video/30709765?title=0&byline=0&portrait=0
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=685
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=393
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=251
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=469
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=657
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/history/?C=S;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=251
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/history/?C=S;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=393
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=347
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=19
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?cat=1&paged=2
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?cat=1
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=149
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=M;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=647
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/history/?C=D;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=D;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=D;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=745
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=9
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=M;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=133
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=S;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=S;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=D;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=823
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=N;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=543
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/history/?C=N;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=S;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=S;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=431
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&author=1
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&author=3
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&author=5
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&author=7
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&author=9
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=187
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=comments-rss2
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=73
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=165
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?author=5
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?author=3
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?author=9
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?author=7
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=M;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=M;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?author=1
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/cmps248/?C=M;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=685
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=149
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=25
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=M;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=469
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=41
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=187&replytocom=79
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=847
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=S;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=S;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/history/?C=M;O=A
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=101
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=823
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=431
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?page_id=281
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/facebook/PromWeek.swf?=4
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&cat=1
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=847
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=543
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/wp-content/themes/san-kloud/style.css?v=1.0.5
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=745
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?paged=2
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=165
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/EditorGUI/?C=N;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/users.soe.ucsc.edu/~mccoyjo/files/?C=N;O=D
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=295
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=657
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=19
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?p=295
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=187
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/player.vimeo.com/video/40632382?title=0&byline=0&portrait=0
producer/ARCHIVEIT-1023-NONE-3625-20131029011626627-00000-wbgrp-crawl050.us.archive.org-6440/promweek.soe.ucsc.edu/?feed=rss2&p=647

Counts by collection of '%?%=%'

SELECT COUNT(f.id)
FROM inv_files f
WHERE f.billable_size > 0
AND f.pathname LIKE '%?%'

824985

Objects by Collection

SELECT c.mnemonic, COUNT(distinct o.ark)
FROM inv_objects o,
inv_files f,
inv_collections c,
inv_collections_inv_objects co
WHERE f.id IN (
SELECT f.id
FROM inv_files f
WHERE f.billable_size > 0
AND f.pathname LIKE '%?%=%')
and f.inv_object_id=o.id
AND co.inv_object_id=o.id
AND c.id=co.inv_collection_id
GROUP BY c.id

----------------------
"mnemonic"  "COUNT(distinct o.ark)"
"ucm_lib_clark" "1"
"ucr_lib_wrca_laaqua"   "7"
"ucsc_lib_promweek" "1"
"ucr_lib_nuxeo" "4548"
"ucsf_lib_stantonglantz"    "241"
"ucm_lib_acm"   "2"
"ucm_lib_mclean"    "40"
"ucm_lib_ucce_ventura"  "1524"
"ucm_lib_ucce_humboldt" "3294"
"ucm_lib_ucce_merced"   "1819"
"uci_lib_sca_ua"    "24"
"uci_lib_sca_se_asian"  "1"
"uci_lib_sca_sc"    "54"
"uci_lib_sca_crit"  "2"
"ucm_lib_ucce_sbslo"    "1674"
"ucsf_lib_wan"  "92"
"ucm_lib_ucce_anr"  "236"
"uci_lib_sca_upub"  "15"
"ucm_lib_elowe" "2767"
"ucm_lib_ucce_fresno"   "2899"
"ucm_lib_ucce_mariposa" "1745"
"ucm_lib_ucce_madera"   "3467"
"ucm_lib_ucce_modoc"    "1132"
"ucm_lib_ucce_sd"   "110"
"ucm_lib_ucce_sjo"  "1791"
"ucm_lib_ucce_syu"  "68"
"ucm_lib_ucce_tulare"   "393"
"uci_lib_sca_orange"    "1"
"ucsf_ctp_archive"  "67"
"ucr_lib_eatonfz"   "1193"
"ucsf_lib_nhprc_gofman" "341"
"ucsf_lib_nhprc_hardgrove"  "97"
"ucsf_lib_nhprc_beyer"  "9"
"ucsf_lib_nhprc_fraiberg"   "508"
"ucsf_lib_nhprc_thelander"  "80"
_________________________________________
By file
SELECT c.mnemonic, COUNT(distinct f.id)
FROM inv_objects o,
inv_files f,
inv_collections c,
inv_collections_inv_objects co
WHERE f.id IN (
SELECT f.id
FROM inv_files f
WHERE f.billable_size > 0
AND f.pathname LIKE '%?%=%')
and f.inv_object_id=o.id
AND co.inv_object_id=o.id
AND c.id=co.inv_collection_id
GROUP BY c.id

"mnemonic"  "COUNT(distinct f.id)"
"ucm_lib_clark" "19"
"ucr_lib_wrca_laaqua"   "271"
"ucsc_lib_promweek" "87"
"ucr_lib_nuxeo" "117140"
"ucsf_lib_stantonglantz"    "241"
"ucm_lib_acm"   "2"
"ucm_lib_mclean"    "60"
"ucm_lib_ucce_ventura"  "45175"
"ucm_lib_ucce_humboldt" "43723"
"ucm_lib_ucce_merced"   "73807"
"uci_lib_sca_ua"    "73"
"uci_lib_sca_se_asian"  "3"
"uci_lib_sca_sc"    "76"
"uci_lib_sca_crit"  "6"
"ucm_lib_ucce_sbslo"    "54175"
"ucsf_lib_wan"  "2702"
"ucm_lib_ucce_anr"  "7643"
"uci_lib_sca_upub"  "27"
"ucm_lib_elowe" "8301"
"ucm_lib_ucce_fresno"   "83429"
"ucm_lib_ucce_mariposa" "25681"
"ucm_lib_ucce_madera"   "60399"
"ucm_lib_ucce_modoc"    "39601"
"ucm_lib_ucce_sd"   "3885"
"ucm_lib_ucce_sjo"  "88005"
"ucm_lib_ucce_syu"  "8336"
"ucm_lib_ucce_tulare"   "70227"
"uci_lib_sca_orange"    "3"
"ucsf_ctp_archive"  "134"
"ucr_lib_eatonfz"   "19434"
"ucsf_lib_nhprc_gofman" "22129"
"ucsf_lib_nhprc_hardgrove"  "11085"
"ucsf_lib_nhprc_beyer"  "393"
"ucsf_lib_nhprc_fraiberg"   "25387"
"ucsf_lib_nhprc_thelander"  "12057"

CDLUC3 / mrt-doc

Store Object "reset" #1774