Closed jrwdunham closed 5 years ago
@jhsimpson @sromkey
Three development branches in three different repos have been created to deal with this issue:
The most significant of these is the SS one, which introduces the import_aip
Django management command which can import an existing (vintage) AIP into a running Storage Service instance, resulting in a compressed AIP with a pointer file. See import_aip
The general methodology employed here was to use import_aip
to import exemplars of the various types of vintage AIP (i.e., types 0.0.1
, 1.0.0
, and 1.0.1
) and then perform various actions and tests—in particular AIP re-ingest—to test whether Archivematica is able to handle these AIPs correctly.
0.0.1
: uncompressed, no pointer file, METS namespacedexemplar AIP: new_uncompressed.tar.gz
import_aip
command in a Docker-Compose environment:
$ make manage-ss ARG='import_aip /home/archivematica/new_uncompressed.tar.gz --decompress-source --compression-algorithm="7z with bzip"'
result: AIPs of this type appear to be functional in Archivematica after importation. Evidence:
Unpackaging the imported AIP and diffing the resulting directory with the original directory shows they are identical:
$ diff -rq CVA1426-0fdedcc1-599d-4f6e-b132-4b7bc0355b1f/ tmp/CVA1426-0fdedcc1-599d-4f6e-b132-4b7bc0355b1f
Initiating a full re-ingest on the imported AIP from the Storage Service succeeds.
1.0.0
: compressed, no pointer file, METS not namespacedexemplar AIP: AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z
import_aip
command in a Docker-Compose environment:
$ make manage-ss ARG='import_aip /home/archivematica/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z --decompress-source --compression-algorithm="7z with bzip"'
result: AIPs of this type are not functional in Archivematica after importation. This appears to be due to the older METS files which, in addition to lacking namespaces, have other differences that make Archivematica choke. Evidence:
Attempts to perform a metadata re-ingest on the imported AIP from the Storage Service have failed at the micro-service Generate METS.xml document
. Issues discovered:
Issue 1: The METS file of AIP AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff
lacks a <mets:metsHdr>
element. metsrw at branch dev/issue-24-handle-old-aips was modified to be able to handle this.
Issue 2: The METS file of AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff
has a strange LABEL
value in its <structMap>
. Notice that the UUID of the
AIP (bdcb560d-7ddd-4c13-8040-1e565b4eddff
) is repeated:
Issue 3: The METS file of AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff
lacks <mets:div>
elements for files—it only has <mets:fptr>
elements. Compare this namespace-less <mets:structMap>
to the following namespaced one:
TODO Next: continue to modify mets-reader-writer
and AM::clientScripts/archivematicaCreateMETSReingest
so that they can handle these older AIPs, using MCPClient/tests/test_reingest_mets_no_namespaces.py to confirm correct behaviour.
1.0.1
: compressed, no pointer file, METS namespacedexemplar AIP: CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32.7z
import_aip command in a Docker-Compose environment:
$ make manage-ss ARG='import_aip /home/archivematica/CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32.7z --decompress-source --compression-algorithm="7z with bzip"'
result: AIPs of this type appear to be functional in Archivematica after importation. Evidence:
Unpackaging the imported AIP and diffing the resulting directory with the original directory shows they are identical:
$ diff -rq CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32 CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32-2
Initiating a full re-ingest on the imported AIP from the Storage Service succeeds.
@hakamine, I've updated JD's branches with a number of fixes. The main changes to support old METS:
<mets:fptr>
lacks a <mets:div>
wrapperThis is the latest re-ingested METS (using AM68's AIP): METS.xml.
There is a known issue with the validator:
[...]
ERROR ON LINE 2457: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': [facet 'enumeration'] The value '2.2' is not an element of the set {'2.0', '2.1'}.
ERROR ON LINE 2457: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': '2.2' is not a valid value of the atomic type '{info:lc/xmlns/premis-v2}versionSimpleType'.
ERROR ON LINE 2471: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': [facet 'enumeration'] The value '2.2' is not an element of the set {'2.0', '2.1'}.
ERROR ON LINE 2471: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': '2.2' is not a valid value of the atomic type '{info:lc/xmlns/premis-v2}versionSimpleType'.
I haven't found a way to combine in the same document elements using premis-v2-1.xsd
and premis-v2-2.xsd
. The validator seems to associate the namespace (info:lc/xmlns/premis-v2
) only to the schema location appearing first.
CC @hakamine @evelynPM
Hi @sevein I have a question re the error output shown in the previous comment. Do the errors show in the output of the import_aip
command or the output of a different tool? I tried the import command on the same AIP and got a "successfully imported" message (no error messages). I am using: AM: dev/issue-24-handle-old-aips 20e2977 and SS: dev/issue-24-handle-old-aips f3fd8dd
Hi @hakamine! It's good news that it's working for you. The error mentioned above is only seen during the validation of the METS document generated by reingest of the imported AIP.
The validation issue has not been solved yet, unfortunately. We're still trying to find the best way to go. The root cause seems to be that you can only associate a XML prefix to a namespace once. E.g. if premis
is used once and it points to premis-v2-1.xsd
then that schema is used by the validator at all times regardless the fact that we are declaring different namespaces later on for the same prefix. So when we combine premis v2.1 + v2.2 (or v2.1 + v3.x as https://github.com/archivematica/Issues/issues/370 suggests), the validator errors out. I've thought maybe we could use different prefixes, e.g. premis
and premis3
, but that may not be a good idea.
When trying to index the AM68 AIP, getting the following error:
...
Rebuilding AIP UUID bdcb560d-7ddd-4c13-8040-1e565b4eddff
Processing AIP bdcb560d-7ddd-4c13-8040-1e565b4eddff
Deleting AIP bdcb560d-7ddd-4c13-8040-1e565b4eddff from aips/aip and aips/aipfile.
Command to run: ['unar', '-force-overwrite', '-o', '/tmp/tmpGzBMkl', '/mnt/siptransfer01-cvan110/aip
store-test-g543/bdcb/560d/7ddd/4c13/8040/1e56/5b4e/ddff/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z
', 'AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff/data/METS.bdcb560d-7ddd-4c13-8040-1e565b4eddff.xml']
/mnt/siptransfer01-cvan110/aipstore-test-g543/bdcb/560d/7ddd/4c13/8040/1e56/5b4e/ddff/AM68-bdcb560d-
7ddd-4c13-8040-1e565b4eddff.7z: 7-Zip
AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff/data/METS.bdcb560d-7ddd-4c13-8040-1e565b4eddff.xml (715
76 B)... OK.
Successfully extracted to "/tmp/tmpGzBMkl/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff".
Removed FITS output from METS.
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/__init__.py", line 354, in execute_from_command_line
utility.execute()
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/__init__.py", line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/base.py", line 394, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/base.py", line 445, in execute
output = self.handle(*args, **options)
File "/opt/archivematica/archivematica/src/dashboard/src/main/management/commands/rebuild_elastics
earch_aip_index_from_files.py", line 295, in handle
delete_existing_data=options['delete'],
File "/opt/archivematica/archivematica/src/dashboard/src/main/management/commands/rebuild_elastics
earch_aip_index_from_files.py", line 169, in processAIPThenDeleteMETSFile
size=aip_info[0]['size'],
File "/usr/lib/archivematica/archivematicaCommon/elasticSearchFunctions.py", line 425, in index_ai
p
mets_created_attr = mets_hdr.get('CREATEDATE')
AttributeError: 'NoneType' object has no attribute 'get'
It seems that the script is looking for an attribute that is not present in the METS file
@hakamine thanks for the report. I've just added a new commit to handle that scenario where metsHdr
is missing. See https://github.com/artefactual/archivematica/commit/ce1579f2b4c677b3f25e4e44deefa5512aac00d2.
Thank you @sevein for the fix. I tested it and the indexing of management the AIP "AM68" (using the custom management command rebuild_elasticsearch_aip_index_from_files
) finished without errors this time. However, I am getting errors when trying to download individual files from the AIP from the archival storage tab:
When I try to download any of the files, getting an internal server error, the dashboard log shows a message like:
ERROR 2019-01-07 23:11:02 django.request:base:handle_uncaught_exception:256: Internal Server Error: /archival-storage/download/aip/file/431913ba-4379-4373-8798-cc5f2b9dd769/
Traceback (most recent call last):
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/archivematica/archivematica/src/dashboard/src/components/archival_storage/views.py", line 318, in aip_file_download
file = models.File.objects.get(uuid=uuid)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
self.model._meta.object_name
DoesNotExist: File matching query does not exist.
(not sure if this could be caused by the "vintage" METS or by a bug in the reindex command script that affects all AIPs. If you think it could be the latter let me know and I'll open a separate github issue)
Thanks, I'll take a look!
Progress update!
Known issues:
@sevein would it be ok if I rebase AM dev/issue-24-handle-old-aips on top of current qa/1.x ? I would like to check if the ES upgrade changes fix the problem with the indexing of AIP files.
Please do! Thanks.
It looks like there is something in AM dev/issue-24-handle-old-aips break indexing of AIP files. I did the following test for both AM qa/1.x (c6396b9) and dev/issue-24-handle-old-aips (d4ac060) branches:
rebuild_elasticsearch_aip_index_from_files
management command for an ingested AIP containing the "Images" sampledata directoryWhile the management command finishes without errors for both branches, the resulting aipfiles
index in branch AM dev/issue-24-handle-old-aips
has all the FILEUUID
fields with value null
. This causes an internal server error when the dashboard's Archival Storage tab when "Show files" is checked, dashboard log is:
ERROR 2019-02-09 00:20:15 django.request:base:handle_uncaught_exception:256: Internal Server Error: /archival-storage/search/
Traceback (most recent call last):
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/archivematica/archivematica/src/dashboard/src/components/archival_storage/views.py", line 180, in search
'page': page_data,
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/shortcuts.py", line 67, in render
template_name, context, request=request, using=using)
...
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/template/defaulttags.py",
line 493, in render
url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app)
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/urlresolvers.py", lin
e 578, in reverse
return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/urlresolvers.py", lin
e 495, in _reverse_with_prefix
(lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'components.archival_storage.views.aip_file_download' with arguments '(None,)' and keyword arguments '{}' no
t found. 1 pattern(s) tried: ['archival-storage/download/aip/file/(?P<uuid>[\\w]{8}(-[\\w]{4}){3}-[\\w]{12})/$']
aipfiles index is attached (dump produced with, e.g.,curl -o search_aipfiles_reindex_devissue24.json http://localhost:9200/aipfiles/_search?pretty=true&q=*:*
)(for comparison, the aipfiles index obtained with branch qa/1.x is also attached.
search_aipfiles_reindex_devissue24.json.zip
search_aipfiles_reindex_qa1x.json.zip
@sevein I added some commits to AM branch dev/issue-24-aip-index-premis2-fallback (which is based off dev/index-24-handle-old-aips), in order to try to fix the indexing issues mentioned in the previous comment. So far it seems to be working (I'll continue testing with other AIPs). Please let me know if the code looks good and I'll merge these commits to branch dev/index-24-handle-old-aips
It looks we are getting closer to the goal! Should we create PRs for the AM/SS/metsrw dev/issue-24-handle-old-aips branches?
Update 1: while the index worked for the "images" sampledata, I am having some problems with AIP AM68 files.
Update 2: the problem with the AM68 AIP is detailed in https://github.com/archivematica/Issues/issues/504
Update 3: added a commit to dev/issue-24-aip-index-premis2-fallback that fixes https://github.com/archivematica/Issues/issues/504
Update 4: tested the AIP import and reindex management commands with a few more AIPs as shown in the screenshot below. Both the import and reindex commands completed without error messages for all of these tested. However, there is an issue with the "Date stored" value in some AIPs, instead of the date they were originally ingested, it shows the date of the reindex (red box in the screenshot)
This occurs for AIPs in group G5 (AIPs) (ingested between 2011/11-2012/09, no namespaces in the METS file, AIPs "AM68" "CVA4-6-7" "AM1553-MI-515" in the screenshot ). However AIPs in group G4 (ingested in 2014, no namespaces in the METS file, AIP "1970-03-03" "1979-09-13_SPEC" in the screenshot) show the correct value for the date.
Taking a look at the METS files, in the G5 AIPs the metsHdr CREATEDATE is not present (i.e., the AM version used to ingest these AIPs didn't include it in the METS file), so the reindex script just uses the current date (the date the reindex command runs) in the
aips
index created
field. In conclusion, this is not a bug of the reindex script (it's a limitation caused by lack of information in the METS files)
@hakamine nice job on dev/issue-24-aip-index-premis2-fallback, thanks for your comment and all the updates. It's all working nicely locally so I'd suggest to update dev/index-24-handle-old-aips with all your changes.
@hakamine, FYI Evelyn noticed that after reingest we have metsHdr
created as follows:
<mets:metsHdr CREATEDATE="2019-02-20T19:57:40"/>
It may be preferably to use LASTMODDATE instead (see https://github.com/artefactual-labs/mets-reader-writer/blob/1f4b7fdbb37512f2c7c42a750b9f08bc73738d9f/metsrw/mets.py#L183-L192), but we won't be addressing that now since it's not a deal breaker.
Thank you @sevein, dev/issue-24-handle-old-aips has been updated (and deleted dev/issue-24-aip-index-premis-fallback)
Update:
@hakamine can you take a look at this and change the label to Done if you feel confident that this is resolved?
Tested on AM qa/1.x 50affa (1.10.0-rc.1) SS qa/0.x fbdf31 (0.15.0-rc.1). Import and reindex of CVA old AIPs is working.
Please describe the problem you'd like to be solved.
Sometimes an institution has been creating AIPs for a long time using various versions of Archivematica and the Storage Service and the institution wants to make sure that those AIPs are usable by their modern (currently installed) Archivematica version, e.g., in order to perform additional preservation actions like adding metadata.
Without modifying the content of such older AIPs, they may need to be compressed and given pointer files that document that compression.
Old AIPs may also have METS files that lack namespaces in their XML elements. It is unclear whether a modern Archivematica pipeline knows how to read such AIPs. Archivematica's METS interpretation functionality (in particular mets-reader-writer) may need to be modified in order to handle such AIPs.
Old AIPs may have accidentally been deleted from a Storage Service database. Their database tables in the SS db may need to be reconstructed from their METS files.
Similarly, old AIPs may have been created on a pipeline that has since been destroyed. Such AIPs may need their SS and MCP (pipeline) database tables reconstructed.
Describe the solution you'd like to see implemented.
I would like to see a case study of the techniques used (or attempted) to make Archivematica flexible and accommodating to various types of vintage AIP. I expect AIP re-ingest may be useful in compressing and generating pointer files for older AIPs, but API calls, Django management commands, and bespoke import/modernize scripts may all be necessary. I expect to see examples of using the AM and SS GUIs and APIs to interact with older AIPs and to discover whether older AIPs can be re-ingested.
Additional context
Imagine you had this:
How would you get to this:?