Import serialized BagIt bags

mikejritter commented 4 years ago

Resolves: https://jira.lyrasis.org/browse/FCREPO-3205

Adds classes to deserialize zip, tar, and gzip BagIt bags

New Dependencies

apache tika: In trying out various ways of reading the content types for files, I found tika to be the most reliable. Using the standard library ended up with some inconsistencies between operating systems (zip read as application/x-compressed-zip on Windows, application/zip on linux). The only issue was that it is precise in the content type it returns, so I did coercion of similar types in the SerializationSupport to map types like gtar and x-gtar to tar and x-tar.
commons-compress: I saw some apache commons libraries used in other projects, so I thought this was a safe bet for handling extraction of tar and gzip archives. I ended up using it for zip extraction as well to keep everything similar.

Notes

I chose not to overwrite any files if they exist, opting to log a warning. If they are not part of the same export the bag should fail to validate later.
Initially I was getting the bag name from matching the expected extension to what was given. e.g. the regex \\.zip for example-bag.zip was used in order to get only example-bag. I that this might not always be the case (maybe a gzipped tarball is just named .tar) so I switched this to the current implementation of getting the last index of a .. I'm still not 100% on this, though.
On that topic, I chose to have the GZipBagDeserializer only perform deflation of the gzip file. This was done in case there was not a tarball in the underlying file... though I'm not sure if this is the flexibility we need. There would likely be an exception thrown either way, so the GZipBagDeserializer could be made to look like the other BagDeserializers fairly easily.
There's a bit of shared code between the archive extraction - I didn't feel comfortable pulling it all out into a single class yet. It's not really any issue yet in my opinion, but just something I wanted to note.
Finally... when the filename in the archive do not match the filename of the archive (e.g. test-bag/ and sample-bag.tar), the BagDeserializers will return an incorrect path. This can be fixed before the PR is merged if desired.

Testing

Start a fresh fedora repository
Create a binary

Create a bag-config.yml for export

bag-info.txt:
Source-Organization: fcrepo-import-export
Organization-Address: localhost
External-Description: Sample bag export for fcrepo-3205

Run the exporter

java -jar target/fcrepo-import-export-0.4.0-SNAPSHOT.jar --mode export --resource http://localhost:8080/fcrepo/rest --dir fcrepo-3205 --binaries --bag-profile default --bag-config bag-config.yml --user fedoraAdmin:secret3

Serialize the exported bag in your format of choice

zip -r fcrepo-3205.zip fcrepo-3205
tar cf fcrepo-3205.tar fcrepo-3205
tar czf fcrepo-3205.tar.gz fcrepo-3205

Remove the deserialized bag
```
rm -r fcrepo-3205
```
Start a fresh fedora repository

Run the importer on the serialized bag

java -Dfcrepo.log.importexport=DEBUG -jar target/fcrepo-import-export-0.4.0-SNAPSHOT.jar --mode import -r http://localhost:8080/fcrepo/rest --dir fcrepo-3205.tar --binaries -g default -G bag-config.yml -u fedoraAdmin:secret3

dbernstein commented 4 years ago

@mikejritter : If it isn't too much trouble I think it is probably a good idea to do this before I test it:

Finally... when the filename in the archive do not match the filename of the archive (e.g. test-bag/ and sample-bag.tar), the BagDeserializers will return an incorrect path. This can be fixed before the PR is merged if desired.

mikejritter commented 4 years ago

@dbernstein I think there are a few additional updates for this for all the tests to pass from the changes #139 brought in. I'll get changes for that tomorrow in addition to seeing if I can get in a fix for the issues you commented on.

mikejritter commented 4 years ago

@dbernstein I made adjustments according to your comments and also updated the BagDeserializers to prioritize the name of the path in the archive. I opted to have it fallback to the filename minus the extension if one is not found, which felt like a sane default.

The BagDeserializerTest was also updated to check for this mismatch, and while I was updating it I made it a Parameterized test since all the tests were the same.

dbernstein commented 4 years ago

@mikejritter : I ran the test, but was running into the following error (after modifying the tar commands - I changed them from tar xf ... to tar cf ... and tar xzf ... to tar czf ...: Here's the the tar import: (tar.gz file also failed for similar reasons).

Daniels-MBP$ java -Dfcrepo.log.importexport=DEBUG -jar ~/code/fcrepo-import-export/target/fcrepo-import-export-0.4.0-SNAPSHOT.jar --mode import -r http://localhost:8080/rest --dir fcrepo-3205.tar.gz --binaries -g default -G bag-config.yml -u fedoraAdmin:fedoraAdmin DEBUG 21:56:43.704 (ArgParser) Command line argments weren't valid for specifying a config file. DEBUG 21:56:44.114 (SerializationSupport) /Users/danielbernstein/tmp/btr/fcrepo-3205.tar.gz: application/gzip ERROR 21:56:44.119 (ImportExportDriver) Error performing import/export: BagProfile does not allow application/gzip. Accepted serializations are: application/tar DEBUG 21:56:44.122 (ImportExportDriver) Stacktrace: java.lang.RuntimeException: BagProfile does not allow application/gzip. Accepted serializations are: application/tar at org.fcrepo.importexport.common.SerializationSupport.deserializerFor(SerializationSupport.java:132) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] at org.fcrepo.importexport.importer.Importer.loadBagProfile(Importer.java:193) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] at org.fcrepo.importexport.importer.Importer.(Importer.java:170) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] at org.fcrepo.importexport.ArgParser.parse(ArgParser.java:448) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] at org.fcrepo.importexport.ImportExportDriver.run(ImportExportDriver.java:57) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] at org.fcrepo.importexport.ImportExportDriver.main(ImportExportDriver.java:47) ~[fcrepo-import-export-0.4.0-SNAPSHOT.jar:na] Daniels-MBP$

mikejritter commented 4 years ago

@dbernstein there are two things I think we can do for this:

Add x-tar/zip/gzip/x-gzip to the supported serialization types for the default bag profile
Update the mappings in SerializationSupport to have the x- prefixed content types go to the "normal" form, e.g. application/x-tar -> application/tar. Since some of the bag profiles were specifying things like application/tar and application/x-tar I had kept them separate in the mapping, but I don't see any harm in unifying them.

Otherwise I think I had been testing with the beyondtherepository profile, but I do think it would be good to at least update the default profile with more supported types.

dbernstein commented 4 years ago

@mikejritter : were you going to add all the supported types that you had in beyond the repository to the default? ie

"application/x-tar",
"application/x-gzip",
"application/x-7z-compressed"

dbernstein commented 4 years ago

Once that's in I'll merge it (as I tested successfully with the btr profile.

mikejritter commented 4 years ago

@dbernstein I updated the default profile to have all serialization formats, it should be good to go. I have a note for myself about further coercion for potentially doing x-tar -> tar, and I think that's something which should stay as a note for now since the profile has the content types enumerated.

fcrepo-exts / fcrepo-import-export