newmanld
commented
8 years ago Script available for checking all objects and verifying whether there are any missing datastreams - datastreams Fedora claims it has but are not found (due to this bug)
Quoting from J. Echols on same thread on hydra-tech:
https://gist.github.com/jechols/c85efdca21e06d54a1f1
not production-quality code, but it does the job.
A few caveats:
- This is VERY brute-force. And very slow. I didn't even attempt to figure out Fedora's bucketing algorithm, so I just scan all files. This is fine for doing a full scan, but terrible if you need to just quickly check a single asset.
- For us, this is all run on an rsync of the data. If you run this live, I have no idea how it'll affect your app's performance.
- I made the system cache the lists of files when I built the script so I didn't have to keep re-scanning the filesystem. Obviously that's terrible for an ongoing process, so if you used it weekly or something, you'd want to fry the "objectrefs.txt" and "fedora-data-files.txt" files first.
- For some reason I spew "." for every datastream being scanned, as if anybody wants a couple million dots filling up their screen. It's best to redirect stdout to /dev/null and only pay attention to stderr. I am trying to re-run it now, but I'm pretty sure the final echo on line 25 ensures you'll get all the info you need if you just capture stderr.
hortongn
Per this thread in Hydra-Tech. https://groups.google.com/forum/#!topic/hydra-tech/RuFv96JJX3I
The thread reports the possibility of data loss in 3.7. Ben sums it up: Forgive the soapboxing, but I really cannot stress enough the importance of getting onto 3.8.1. It requires JDK 8, that's probably your biggest hurdle.
Description of the 3.7.x bug: This is might be due to running parallel updates to objects on Fedora < 3.8.1, where an object locking flaw could lead to phantom DS updates when multiple updates to the same datastream were being processed simultaneously.
For more detail see: https://jira.duraspace.org/browse/FCREPO-1284