Creating binary placeholders for binaries

[escowles]

Fixes:

• https://jira.duraspace.org/browse/FCREPO-2198
• https://jira.duraspace.org/browse/FCREPO-2201
• https://jira.duraspace.org/browse/FCREPO-2219
• https://jira.duraspace.org/browse/FCREPO-2231

I am able to ingest the plantpatents dataset with this branch.

[awoods]

I am getting the following build error when running mvn clean install on this PR.

testReferences(org.fcrepo.importexport.integration.ImporterIT)  Time elapsed: 0.727 sec  <<< ERROR!
java.lang.RuntimeException: Error while importing /home/awoods/programming/java/apps/fedora/futures/fcrepo-import-export/target/test-classes/sample/reference/fcrepo/rest/linkTo.ttl (400): Unable to set existing single-valued property "ns001:relation_ref" on node "/linkTo" in workspace "default" using multi-value setter methods
Unable to set existing single-valued property "ns002:hasFile_ref" on node "/linkTo" in workspace "default" using multi-value setter methods
        at org.fcrepo.importexport.importer.Importer.importFile(Importer.java:163)
        at org.fcrepo.importexport.importer.Importer.lambda$importDirectory$0(Importer.java:120)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.fcrepo.importexport.importer.Importer.importDirectory(Importer.java:120)
        at org.fcrepo.importexport.importer.Importer.run(Importer.java:112)
        at org.fcrepo.importexport.integration.ImporterIT.testReferences(ImporterIT.java:136)

[ajs6f]

/home/awoods/programming/java: I think it's really sweetly old-fashioned the way you parcel out projects into directories based on language. It's like having an old-time malted milk at a downtown soda fountain while waiting for the picture show.

[escowles]

@awoods Travis and I aren't seeing that error when we build — but I remember @ruebot seeing it when trying to load certain datasets with this PR yesterday. I was planning to continue working on getting the datasets to load, so I can see if I can trigger that error and work to resolve it in this PR. Does anybody else see this problem when building this PR?

[ruebot]

@awoods @escowles builds fine for me. The plant patent dataset exports and imports. I have an error -- that'll I'll create a ticket for -- on lubm and the hydra dataset. 10k is still importing after 6hrs+, and I'm going to let that run until I head over the campus in an hour or so to see what happens.

[awoods]

Given our lack of dependency on fcrepo4, I would not think it would matter; but my local build of fcrepo4 is up-to-date with master...

[ruebot]

@awoods I'm running these tests with 4.6.0 one-click.

[escowles]

@awoods I just built fcrepo4 master, and still don't see the error. The config is pulling down 4.6.0 w/webacl, so I doubt the local instance of fcrepo4 is the culprit: https://github.com/fcrepo4-labs/fcrepo-import-export/blob/placeholders/pom.xml#L221

But I do wonder why you would see something different.

[awoods]

For comparison sake, here is my tomcat log from the integration test: https://gist.github.com/awoods/f13bddb706bf9714d12c7ada9b0d4f6a

Here is the 400: https://gist.github.com/awoods/f13bddb706bf9714d12c7ada9b0d4f6a#file-localhost_access_log-2016-09-22-txt-L7

[bseeger]

fwiw - I can build this PR just fine - with mvn clean install. That's all I've tried though. Would one need to do more to see the error?

[escowles]

@awoods I just pushed another commit that resolves the Unable to set existing single-valued property error I was seeing when trying to import the LUBM02 dataset. I believe the error was being triggered by creating pairtree nodes and then trying to set properties on them.

With this second commit, I can load the LUBM02 dataset without errors. I also noticed that doing HEAD requests to check if the URIs existed is much faster for me than just doing the quasi-conditional-PUTs was: I was seeing the error only after about 14 minutes, but I can now load the LUBM02 dataset is around 8 minutes.

[ruebot]

@escowles @awoods builds for me, and lubm dataset imports fine now :tada:

I'm going to run the Hydra and 10k Indirect containers datasets next.

[ruebot]

@escowles @awoods Hydra and 10k datasets import with no errors!

[ruebot]

@awoods are you cool with me merging this, or are you still having issues?

[ruebot]

Merging this so we can start to wrap up Penn State sprint.