Open mjaddis opened 4 years ago
sromkey
commented
4 years ago @mjaddis I'm curious what you think of the solution being a less verbose METS vs successfully printing the METS in the format we currently have? I'm thinking for example of some of the concepts explored here: https://wiki.archivematica.org/PREMIS/METS_for_scalability
mjaddis
commented
4 years ago @sromkey I think those solutions are complementary rather than alternatives. I'm all in favour of reducing METS verbosity - that benefits all Archivematica users irrespective of the size of the Transfers they might have - which can only be a good thing. I remember discussing many of the things on the wiki page when we were working with you guys on the RDSS project. In my case there's lots of opportunity for deduplication in the METS, so it would undoubtedly help. For example, there's 1.4million digiprovMDs in my METS, so even simple things like saying 'all files passed virus checking' and 'all files are fmt/40' rather than having this repeated 100,000 times would save millions of lines of XML. But this to some extent only kicks the can down the road. It might get to 500,000 files in a Transfer rather than 50,000, but there would still be a limit. That's because all the METS is still in a single file and it eventually gets huge. One of the things I've always thought would be nice is to split the METS file apart into lots of smaller files, for example a METS file for each file in the Transfer as well as a top-level METS backbone that ties them all together. For example, if there was a directory tree that contained METS files that mirrored the directory tree of files I want preserved, then this could hold a hierarchy of METS files that matched my content files. Each one would be a few MBs in size and it would naturally scale. Of course that brings in many other issues, e.g. backwards compatibility with existing AIPs and a load of development work in Archivematica, so is probably something that can't be done that easily.
Expected behaviour The METS file for an AIP should be created and serialised without errors even for datasets with a large number of files or lots of descriptive metadata in dmdSecs, i.e. if the METS is really big.
Current behaviour An error is thrown when the METS is serialised to disk during the Ingest stage of the workflow. The METS appears to be created correctly in memory and it also appears to be written OK to disk (the file is complete and not half-written etc.). However, a serialisation error is thrown and the workflow fails.
The METS file created is approx 3.4GB in size.
If I don't include any metadata then the same failure occurs but with a different error code:
However, if I drop the number of files to 30,000 but still include a metadata.csv (only 60,000 rows!) then it works fine. There's something about the size/complexity of the METS that causes problems when there are a large number of files in the Transfer.
Steps to reproduce I made a Transfer into Archivematica 1.11 on ubuntu 18.04 that had 100,000 files that were organised into approx 100,000 directories. I included a metadata.csv that attaches metadata to both the files and the directories. This had 200,000 rows with each row having 20 fields. The files were small MS Word docs and I added FPR capabilities to convert these to PDF/A. I normalised for preservation and I turned off thumbnail generation.
Your environment (version of Archivematica, operating system, other relevant details)
The job was run on an EC2 instance with 32 cores and 128GB of memory.
I don't think this is a memory issue because the lxml code should throw a separate error if there is a memory allocation problem. There's also 128GB of memory on the machine and I didn't see any messages in syslog etc. saying memory was a problem. The 30,000 file dataset that did work also goes through on a machine with much less memory.
I'm wondering if it's the number of XML nodes that's the issue, or the use of pretty-printing.
I also recognise that the problem above is probably right at the extreme of what people are trying to do with Archivematica. The AIP, had it been finished, would have 200,000 files and a METS file with 60M lines of XML.
Overall, I'm posting this issue just to log what I found in case others see the same at some point rather than expecting a bug-fix anytime soon.
For Artefactual use:
Before you close this issue, you must check off the following: