Closed glenrobson closed 6 months ago
Hi,
I tried running the same migration on the latest version of Fedora (4.1.1) and got a different error. Running the migration this time processed the objects in a different order so I don't know if I've come across another problem before reaching an issue with llgc-id:3143635 (which is the object with lots of versions).
Running the migration on 4.1.1 the first object to be processed is llgc-id:3143637 and the migration log looks as follows:
INFO 00:20:51.862 (InternalIDResolver) No index directory specified. Creating temporary index at "/var/folders/9c/j9t_fxld66jbtyq8bx_kmwwd5k5799/T/tempfile8874507445491749144basedir/index".
INFO 00:20:52.154 (InternalIDResolver) Builidng an index of all the datastreams in "src/test/resources/legacyFS/datastreams"...
INFO 00:20:53.168 (Migrator) Processing "llgc-id:3143637"...
DEBUG 00:20:53.315 (BasicObjectVersionHandler) Considering object llgc-id:3143637 version at 2013-02-12T12:50:39.558Z.
DEBUG 00:20:56.034 (BasicObjectVersionHandler) Considering changed datastream version AUDIT.0
DEBUG 00:20:57.799 (BasicObjectVersionHandler) Considering object llgc-id:3143637 version at 2013-02-12T12:50:39.569Z.
DEBUG 00:20:57.800 (BasicObjectVersionHandler) Considering changed datastream version RELS-EXT.0
DEBUG 00:20:58.945 (BasicObjectVersionHandler) Considering changed datastream version METS.0
DEBUG 00:20:59.527 (BasicObjectVersionHandler) Considering changed datastream version DC.0
DEBUG 00:20:59.592 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/identifier value llgc-id:3143637
DEBUG 00:20:59.592 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/title value [1]
DEBUG 00:20:59.592 (BasicObjectVersionHandler) Considering changed datastream version RIGHTS.0
DEBUG 00:20:59.835 (BasicObjectVersionHandler) Considering changed datastream version POLICY.0
DEBUG 00:21:00.073 (BasicObjectVersionHandler) Considering changed datastream version archive.0
DEBUG 00:21:00.310 (BasicObjectVersionHandler) Considering changed datastream version jp2.0
ERROR 00:21:00.726 (FedoraResourceImpl) error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.fcrepo.client.FedoraException: error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.updateResourceProperties(BasicObjectVersionHandler.java:520)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.updateObjectProperties(BasicObjectVersionHandler.java:232)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.processObjectVersions(BasicObjectVersionHandler.java:164)
at org.fcrepo.migration.handlers.VersionAbstractionFedoraObjectHandler.processObject(VersionAbstractionFedoraObjectHandler.java:95)
at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:67)
at org.fcrepo.migration.foxml11.Foxml11InputStreamFedoraObjectProcessor.processObject(Foxml11InputStreamFedoraObjectProcessor.java:129)
at org.fcrepo.migration.Migrator.run(Migrator.java:101)
at org.fcrepo.migration.Migrator.main(Migrator.java:37)
... 6 more
Caused by: org.fcrepo.client.FedoraException: error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request
at org.fcrepo.client.impl.FedoraResourceImpl.updateProperties(FedoraResourceImpl.java:191)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.updateResourceProperties(BasicObjectVersionHandler.java:517)
... 13 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.144 s
[INFO] Finished at: 2015-05-17T00:21:00+01:00
[INFO] Final Memory: 30M/620M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project migration-utils: An exception occured while executing the Java class. null: InvocationTargetException: org.fcrepo.client.FedoraException: error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
It has loaded the following datastreams:
but it looks like it failed on the coordinates datastream which is a managed ALTO XML file. The analysis for this object is below:
INFO 00:19:34.370 (Migrator) Processing "llgc-id:3143637"...
llgc-id:3143637
Properties
info:fedora/fedora-system:def/model#state = Active
info:fedora/fedora-system:def/model#label = [1]
info:fedora/fedora-system:def/model#ownerId =
info:fedora/fedora-system:def/model#createdDate = 2013-02-12T12:50:39.558Z
info:fedora/fedora-system:def/view#lastModifiedDate = 2013-06-05T16:20:26.464Z
AUDIT version AUDIT.0
RELS-EXT version RELS-EXT.0
METS version METS.0
METS version METS.1
DC version DC.0
RIGHTS version RIGHTS.0
POLICY version POLICY.0
archive version archive.0
jp2 version jp2.0
coordinates version coordinates.0
It seems to have loaded the METS but only the first version (unless I'm missing something from the log).
Should I be adding this to JIRA or the fedora tech list rather than github?
Thanks
Glen
here is the relevant fedora (4.1.1) log for the error above:
INFO 00:20:53.570 (FedoraLdp) HEAD for: migrated-fedora3/llgc-id/31/43/63/7
INFO 00:20:54.192 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7'
INFO 00:20:54.358 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7'
INFO 00:20:56.083 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/AUDIT'
INFO 00:20:56.178 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/AUDIT/fcr:metadata'
INFO 00:20:56.563 (FedoraLdp) PATCH for 'migrated-fedora3/llgc-id/31/43/63/7/AUDIT/fcr:metadata'
INFO 00:20:57.273 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/AUDIT/fcr:metadata'
INFO 00:20:57.679 (FedoraVersioning) Request to add version 'imported-version-0' for 'migrated-fedora3/llgc-id/31/43/63/7'
INFO 00:20:57.861 (FedoraLdp) HEAD for: migrated-fedora3/model/sc/if/-n/ew/sp/ap/er/s-/pa/ge
INFO 00:20:57.936 (FedoraLdp) PUT resource 'migrated-fedora3/model/sc/if/-n/ew/sp/ap/er/s-/pa/ge'
INFO 00:20:58.024 (FedoraLdp) GET resource 'migrated-fedora3/model/sc/if/-n/ew/sp/ap/er/s-/pa/ge'
INFO 00:20:58.197 (FedoraLdp) HEAD for: migrated-fedora3/llgc-id/31/43/63/6
INFO 00:20:58.248 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/6'
INFO 00:20:58.307 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/6'
INFO 00:20:58.396 (FedoraLdp) HEAD for: migrated-fedora3/collection/sc/if/_a/pn
INFO 00:20:58.451 (FedoraLdp) PUT resource 'migrated-fedora3/collection/sc/if/_a/pn'
INFO 00:20:58.503 (FedoraLdp) GET resource 'migrated-fedora3/collection/sc/if/_a/pn'
INFO 00:20:58.592 (FedoraLdp) HEAD for: migrated-fedora3/collection/sc/if
INFO 00:20:58.672 (FedoraLdp) HEAD for: migrated-fedora3/collection/di/gi/ti/sa/ti/on
INFO 00:20:58.728 (FedoraLdp) PUT resource 'migrated-fedora3/collection/di/gi/ti/sa/ti/on'
INFO 00:20:58.884 (FedoraLdp) GET resource 'migrated-fedora3/collection/di/gi/ti/sa/ti/on'
INFO 00:20:58.985 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/METS'
INFO 00:20:59.085 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/METS/fcr:metadata'
INFO 00:20:59.252 (FedoraLdp) PATCH for 'migrated-fedora3/llgc-id/31/43/63/7/METS/fcr:metadata'
INFO 00:20:59.402 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/METS/fcr:metadata'
INFO 00:20:59.635 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/RIGHTS'
INFO 00:20:59.714 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/RIGHTS/fcr:metadata'
INFO 00:20:59.884 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/POLICY'
INFO 00:20:59.968 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/POLICY/fcr:metadata'
INFO 00:21:00.128 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/archive'
INFO 00:21:00.206 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/archive/fcr:metadata'
INFO 00:21:00.386 (FedoraLdp) PUT resource 'migrated-fedora3/llgc-id/31/43/63/7/jp2'
INFO 00:21:00.459 (FedoraLdp) GET resource 'migrated-fedora3/llgc-id/31/43/63/7/jp2/fcr:metadata'
INFO 00:21:00.603 (FedoraLdp) PATCH for 'migrated-fedora3/llgc-id/31/43/63/7'
Set the debug to trace and added a printStackTrace to BasicObjectVersionHandler.updateResourceProperties exception handler and got the following:
TRACE 01:01:29.869 (BasicObjectVersionHandler) SPARQL: PREFIX fedoraaccess: <http://fedora.info/definitions/1/0/access/>
PREFIX fedora3model: <info:fedora/fedora-system:def/model#>
PREFIX dcterms: <http://purl.org/dc/terms/>
DELETE WHERE
{
<> <http://dev.llgc.org.uk/digitisation/identifiers/handle> ?o0 .
<> <http://www.openarchives.org/OAI/2.0/itemID> ?o1 .
<> fedora3model:hasModel ?o2 .
<> <info:fedora/fedora-system:def/relations-external#isPartOf> ?o3 .
<> <http://dev.llgc.org.uk/digitisation/identifiers/nlw_id> ?o4 .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> ?o5 .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> ?o6 .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> ?o7 .
<> <http://purl.org/dc/elements/1.1/identifier> ?o .
<> <http://purl.org/dc/elements/1.1/title> ?o .
} ;
INSERT DATA {
<> <http://dev.llgc.org.uk/digitisation/identifiers/handle> "10107/3143637" .
<> <http://www.openarchives.org/OAI/2.0/itemID> "oai:llgc.org.uk:llgc-id:3143637" .
<> fedora3model:hasModel <http://localhost:8888/rest/migrated-fedora3/model/sc/if/-n/ew/sp/ap/er/s-/pa/ge> .
<> <info:fedora/fedora-system:def/relations-external#isPartOf> <http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/6> .
<> <http://dev.llgc.org.uk/digitisation/identifiers/nlw_id> "apnag00100001" .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> <http://localhost:8888/rest/migrated-fedora3/collection/sc/if/_a/pn> .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> <http://localhost:8888/rest/migrated-fedora3/collection/sc/if> .
<> <info:fedora/fedora-system:def/relations-external#isMemberOfCollection> <http://localhost:8888/rest/migrated-fedora3/collection/di/gi/ti/sa/ti/on> .
<> <http://purl.org/dc/elements/1.1/identifier> "llgc-id:3143637" .
<> <http://purl.org/dc/elements/1.1/title> "[1]" .
}
ERROR 01:01:30.007 (FedoraResourceImpl) error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request
org.fcrepo.client.FedoraException: error updating resource http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/7: 400 Bad Request
at org.fcrepo.client.impl.FedoraResourceImpl.updateProperties(FedoraResourceImpl.java:191)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.updateResourceProperties(BasicObjectVersionHandler.java:517)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.updateObjectProperties(BasicObjectVersionHandler.java:232)
at org.fcrepo.migration.handlers.BasicObjectVersionHandler.processObjectVersions(BasicObjectVersionHandler.java:164)
at org.fcrepo.migration.handlers.VersionAbstractionFedoraObjectHandler.processObject(VersionAbstractionFedoraObjectHandler.java:95)
at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:67)
at org.fcrepo.migration.foxml11.Foxml11InputStreamFedoraObjectProcessor.processObject(Foxml11InputStreamFedoraObjectProcessor.java:129)
at org.fcrepo.migration.Migrator.run(Migrator.java:101)
at org.fcrepo.migration.Migrator.main(Migrator.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
pasting the sparql into the update box in the Fedora4 web interface and clicking update returns the following error:
Bad Request
Only referenceable nodes may be the value of reference properties
Thanks for all the helpful debugging information! The last statement seems the most helpful in that it's pointing out that the RDF update is bad because it references other resources (in fedora 4) that don't exist. We did some work a couple weeks ago to fix a similar issue, but perhaps there's a case we missed.
Placeholders for the following objects (referenced in RELS-EXT) should have been made when processing the RELS-EXT here.
Could you verify that the referenced objects exist? They will be empty and have no versions if they were made as placeholders to support creating links before they were migrated.
http://localhost:8888/rest/migrated-fedora3/model/sc/if/-n/ew/sp/ap/er/s-/pa/ge http://localhost:8888/rest/migrated-fedora3/llgc-id/31/43/63/6 http://localhost:8888/rest/migrated-fedora3/collection/sc/if/_a/pn http://localhost:8888/rest/migrated-fedora3/collection/sc/if http://localhost:8888/rest/migrated-fedora3/collection/di/gi/ti/sa/ti/on
Thanks for looking into this and for the explanation of the error (the error makes a lot more sense now you've explained it).
All of those links above do resolve to objects and the only one that looks a bit strange is: http://localhost:8888/rest/migrated-fedora3/collection/sc/if which does have the following child: http://localhost:8888/rest/migrated-fedora3/collection/sc/if/_a/pn I've haven't included the collection:scif_apn or collection:scif objects in the foxml directory so I'm guessing this relationship has been created by the SimpleIDMapper class. Would this child/parent relationship cause the problem above?
I was thinking of creating a new MigrationIDMapper that recognises if the characters after : in the PID are not numeric and if so just create the path with the word in it for example the two collections above would be:
http://localhost:8888/rest/migrated-fedora3/collection/scif http://localhost:8888/rest/migrated-fedora3/collection/scif_apn
the vast majority of our content is llgc-id:number which I was going to leave as the current layout (split after two numbers) but our collections and content models have letters after the colon and would look nicer as full words. Do you think that would be a good idea? If the parent child relationship was the problem then this would also fix that.
Thanks
Glen
Your approach for a new MigrationIDMapper is just the sort of customization I expected people to make. I hope it's quite easy to implement.
I've reproduced the error you got by creating a similar RELS-EXT. I'll track the work fixing this bug in our Jira instance here: https://jira.duraspace.org/browse/FCREPO-1547.
I've implemented a new MigrationIDMapper:
https://github.com/fcrepo4-labs/migration-utils/compare/master...glenrobson:nlw_changes
and it seems to have fixed the 400 error. I hope it isn't too lazy but I extended your ID Mapper and changed some of the variables and methods from private to protected so I only needed to add a pidToPath method.
On to another problem which Ill create a new issue for.
Supported pathway is now Fedora 3 to 6. Clearing out stale issues
Hi,
I'm trying to migrate 6 objects from Fedora3 to Fedora4. I've copied the object XML and datastreams onto a Fedora 4 server matching the directory layout from Fedora 3. I've configured the migration-bean.xml to use nativeFoxmlDirectoryObjectSource and legacyIDResolver and when I ran the consoleLoggingStreamingFedoraObjectHandler it gave me the following:
but when I try and run with objectAbstraction instead of consoleLoggingStreamingFedoraObjectHandler I get:
It seems to create the audit datastream but stop at the first RELS-EXT version (Note we have a lot of RELS-EXTs and METS versions in this object).
Sorry if I've missed something obvious. I'm using Fedora version 4.1.0 and the migration utils downloaded yesterday from the master.
Thanks
Glen