Closed DiegoPino closed 4 years ago
Thanks! I just tried my Collection ingest again, after checking off the "do not enforce" option.
Problems:
Yep, the whole thing has crashed because of this. Any advice on bringing it back?
Holiday here. But what so you mean crashed it all? I can’t imagine IMI can totally crash your instance, can you share some apache or drupal logs. Zip file request will keep happening if you are selecting zip file from the source for ds on the object mapping. Select any other
Also, not 500 related, but make sure you delete any previous batch since you are passing fixed PIDs (collections) and you are probably not using the patched islandora batch module branch we have. Still, surprised and curious about that 500 error, other than resetting my google token i had no other issues and was able to ingest a batch spreadsheet with every cmodel we use for testing
El El lun, 25 de may. de 2020 a la(s) 14:34, Diego Pino dpino@metro.org escribió:
Holiday here. But what so you mean crashed it all? I can’t imagine IMI can totally crash your instance, can you share some apache or drupal logs. Zip file request will keep happening if you are selecting zip file from the source for ds on the object mapping. Select any other
El El lun, 25 de may. de 2020 a la(s) 12:39, Brandon Weigel < notifications@github.com> escribió:
Yep, the whole thing has crashed because of this. Any advice on bringing it back?
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/pull/117#issuecomment-633646756, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU7ZZZZ7AZ7KVO3RWNDYCLRTKNKLANCNFSM4NIDQMQQ .
-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)
-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)
Zip file request will keep happening if you are selecting zip file from the source for ds on the object mapping.
Can we have a "no files" option in the list?
But what so you mean crashed it all? I can’t imagine IMI can totally crash your instance
I don't know, but if I go to localhost:8000, I get a 500 error. Can't go to any page in the Vagrant machine, it's all 500 errors. This is after the final stage of trying to create the batch.
you are probably not using the patched islandora batch module branch we have
I am currently on your patched Batch branch.
can you share some apache or drupal logs
Can't access my Drupal logs. Apache logs show this:
[Mon May 25 15:45:25.323158 2020] [:error] [pid 4410] [client 10.0.2.2:62712] PHP Parse error: syntax error, unexpected '<<' (T_SL) in /var/www/drupal/sites/all/modules/islandora_batch/includes/ingest.batch.inc on line 420, referer: http://localhost:8000/multi_importer
[Mon May 25 15:45:51.277855 2020] [:error] [pid 1421] [client 10.0.2.2:62732] PHP Parse error: syntax error, unexpected '<<' (T_SL) in /var/www/drupal/sites/all/modules/islandora_batch/islandora_batch.install on line 362, referer: http://localhost:8000/multi_importer
Checking out the 7.x branch and clearing the batch sets seems to have gotten rid of the 500 error. Will re-test...
So on the Object Properties Mapping, under Remote DS Sources, there is no "none" option. This will need to be added.
I've gone through the process, on this branch, and I'm still getting errors on the ingest. I did make sure that the objects did not already exist and that the batch sets were cleared out.
PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'npc:lanz' for key 'PRIMARY': INSERT INTO {islandora_batch_queue} (id, parent, data, sid) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3); Array ( [:db_insert_placeholder_0] => npc:lanz [:db_insert_placeholder_1] => npc:root [:db_insert_placeholder_2] => O:25:"IslandoraMultiBatchObject":17:{s:11:"*baseName";s:29:"New Object via Multi importer";s:13:"*objectInfo";a:6:{s:6:"cmodel";s:26:"islandora:collectionCModel";s:6:"parent";s:8:"npc:root";s:4:"data";a:63:{i:0;N;i:1;N;i:2;s:26:"islandora:collectionCModel";i:3;s:8:"npc:root";i:4;s:22:"Walter Lanz Collection";i:5;s:8:"npc:lanz";i:6;N;i:7;N;i:8;N;i:9;N;i:10;N;i:11;N;i:12;N;i:13;N;i:14;N;i:15;N;i:16;N;i:17;N;i:18;N;i:19;N;i:20;N;i:21;N;i:22;N;i:23;N;i:24;N;i:25;N;i:26;N;i:27;N;i:28;N;i:29;N;i:30;N;i:31;N;i:32;N;i:33;N;i:34;N;i:35;N;i:36;N;i:37;N;i:38;N;i:39;N;i:40;N;i:41;N;i:42;N;i:43;N;i:44;N;i:45;N;i:46;N;i:47;N;i:48;N;i:49;N;i:50;N;i:51;N;i:52;N;i:53;N;i:54;N;i:55;N;i:56;N;i:57;N;i:58;N;i:59;N;i:60;N;i:61;N;i:62;N;}s:3:"pid";s:8:"npc:lanz";s:13:"parent_cmodel";a:2:{i:0;s:26:"islandora:collectionCModel";i:1;s:30:"fedora-system:FedoraObject-3.0";}s:9:"namespace";s:3:"npc";}s:15:"*currentState";i:1;s:25:"*preprocessorParameters";a:8:{s:4:"type";s:6:"REMOTE";s:11:"source_data";s:2:"60";s:25:"cmodel_source_field_index";s:1:"2";s:14:"cmodel_mapping";a:1:{s:26:"islandora:collectionCModel";a:1:{s:4:"dsid";a:1:{s:4:"rows";a:2:{s:2:"DC";a:1:{s:6:"method";a:1:{s:4:"data";s:6:"xlst|0";}}s:2:"TN";a:1:{s:6:"method";a:1:{s:4:"data";s:0:"";}}}}}}s:13:"object_maping";a:6:{s:10:"pidmap_row";a:2:{s:6:"pidmap";s:1:"5";s:7:"pidtype";i:0;}s:13:"parentmap_row";a:2:{s:9:"parentmap";s:1:"3";s:10:"parenttype";i:1;}s:12:"labelmap_row";a:1:{s:8:"labelmap";s:1:"4";}s:15:"sequencemap_row";a:1:{s:11:"sequencemap";s:1:"1";}s:9:"dsmap_row";a:1:{s:8:"dsremote";s:6:"REMOTE";}s:11:"relsext_row";a:1:{s:12:"relsext_mode";s:3:"NEW";}}s:16:"computed_cmodels";a:1:{s:26:"islandora:collectionCModel";a:4:{s:2:"DC";a:3:{s:2:"id";s:2:"DC";s:4:"mime";a:1:{i:0;s:15:"application/xml";}s:8:"optional";b:0;}s:8:"RELS-EXT";a:3:{s:2:"id";s:8:"RELS-EXT";s:4:"mime";a:1:{i:0;s:19:"application/rdf+xml";}s:8:"optional";b:1;}s:8:"RELS-INT";a:3:{s:2:"id";s:8:"RELS-INT";s:4:"mime";a:1:{i:0;s:19:"application/rdf+xml";}s:8:"optional";b:1;}s:2:"TN";a:3:{s:2:"id";s:2:"TN";s:4:"mime";a:3:{i:0;s:10:"image/jpeg";i:1;s:9:"image/png";i:2;s:9:"image/gif";}s:8:"optional";b:0;}}}s:6:"action";s:6:"ingest";s:4:"data";a:1:{s:7:"headers";a:63:{i:0;s:8:"filename";i:1;s:8:"sequence";i:2;s:6:"cmodel";i:3;s:6:"parent";i:4;s:5:"title";i:5;s:6:"newpid";i:6;s:8:"subtitle";i:7;s:18:"contributor1_given";i:8;s:19:"contributor1_family";i:9;s:17:"contributor1_role";i:10;s:18:"contributor2_given";i:11;s:19:"contributor2_family";i:12;s:17:"contributor2_role";i:13;s:20:"corporatecontributor";i:14;s:25:"corporatecontributor_role";i:15;s:14:"corporatedonor";i:16;s:9:"publisher";i:17;s:15:"publisher_place";i:18;s:11:"date_issued";i:19;s:14:"date_qualifier";i:20;s:11:"description";i:21;s:6:"extent";i:22;s:14:"subject1_topic";i:23;s:14:"subject2_topic";i:24;s:14:"subject3_topic";i:25;s:14:"subject4_topic";i:26;s:14:"subject5_topic";i:27;s:18:"subject_geographic";i:28;s:9:"continent";i:29;s:7:"country";i:30;s:6:"region";i:31;s:5:"state";i:32;s:9:"territory";i:33;s:6:"county";i:34;s:4:"city";i:35;s:11:"citysection";i:36;s:6:"island";i:37;s:4:"area";i:38;s:20:"extraterrestrialarea";i:39;s:11:"coordinates";i:40;s:14:"subject1_given";i:41;s:15:"subject1_family";i:42;s:14:"subject1_title";i:43;s:14:"subject2_given";i:44;s:15:"subject2_family";i:45;s:14:"subject2_title";i:46;s:5:"genre";i:47;s:15:"genre_authority";i:48;s:4:"type";i:49;s:4:"form";i:50;s:17:"internetmediatype";i:51;s:8:"language";i:52;s:20:"physical_description";i:53;s:5:"notes";i:54;s:15:"localidentifier";i:55;s:3:"uri";i:56;s:4:"isbn";i:57;s:3:"doi";i:58;s:16:"physicallocation";i:59;s:16:"related_resource";i:60;s:6:"rights";i:61;s:19:"creativecommons_uri";i:62;s:15:"rightsstatement";}}}s:17:"modsToDcTransform";s:62:"sites/all/modules/islandora_multi_importer/xslt/mods_to_dc.xsl";s:8:"deriveDC";b:1;s:20:"modsCleanUpTransform";s:90:"sites/all/modules/islandora_multi_importer/xslt/islandora_cleanup_mods_extended_strict.xsl";s:9:"cleanMODS";b:0;s:27:"*newFedoraDatastreamClass";s:28:"IslandoraNewFedoraDatastream";s:24:"*fedoraDatastreamClass";s:25:"IslandoraFedoraDatastream";s:21:"*fedoraRelsExtClass";s:22:"IslandoraFedoraRelsExt";s:14:"*datastreams";a:1:{s:8:"RELS-EXT";O:28:"IslandoraNewFedoraDatastream":8:{s:21:"*fedoraRelsIntClass";s:22:"IslandoraFedoraRelsInt";s:31:"*fedoraDatastreamVersionClass";s:32:"IslandoraFedoraDatastreamVersion";s:9:"*copied";b:0;s:10:"repository";O:25:"IslandoraFedoraRepository":6:{s:13:"*queryClass";s:24:"IslandoraRepositoryQuery";s:17:"*newObjectClass";s:24:"IslandoraNewFedoraObject";s:14:"*objectClass";s:21:"IslandoraFedoraObject";s:8:"*cache";O:20:"IslandoraSimpleCache":0:{}s:2:"ri";O:24:"IslandoraRepositoryQuery":1:{s:10:"connection";O:29:"IslandoraRepositoryConnection":11:{s:3:"url";s:28:"http://localhost:8080/fedora";s:7:"cookies";b:1;s:8:"username";s:5:"admin";s:8:"password";s:55:"$S$DdUES95IVRTk1eiCYdJ0Hmb7E8cgmKu2DoF32Ml69u53DAcHrsDJ";s:10:"verifyHost";b:1;s:10:"verifyPeer";b:1;s:7:"timeout";N;s:14:"connectTimeout";i:5;s:9:"userAgent";N;s:15:"reuseConnection";b:1;s:10:"sslVersion";N;}}s:3:"api";O:18:"IslandoraFedoraApi":3:{s:1:"a";O:10:"FedoraApiA":2:{s:13:"*connection";r:213;s:13:"*serializer";O:19:"FedoraApiSerializer":0:{}}s:1:"m";O:19:"IslandoraFedoraApiM":2:{s:10:"connection";r:213;s:10:"serializer";r:228;}s:10:"connection";r:213;}}s:6:"parent";r:1;s:13:"relationships";O:22:"IslandoraFedoraRelsInt":6:{s:10:"*aboutDs";r:203;s:6:"*new";b:0;s:11:"*domCache";N;s:10:"datastream";N;s:13:"*namespaces";a:2:{s:3:"rdf";s:43:"http://www.w3.org/1999/02/22-rdf-syntax-ns#";s:9:"islandora";s:37:"http://islandora.ca/ontology/relsint#";}s:18:"nonMagicAutoCommit";b:1;}s:15:"*datastreamId";s:8:"RELS-EXT";s:17:"*datastreamInfo";a:10:{s:14:"dsControlGroup";s:1:"X";s:7:"dsState";s:1:"A";s:7:"dsLabel";s:46:"Fedora Object to Object Relationship Metadata.";s:13:"dsVersionable";b:1;s:6:"dsMIME";s:19:"application/rdf+xml";s:11:"dsFormatURI";s:43:"info:fedora/fedora-system:FedoraRELSExt-1.0";s:14:"dsChecksumType";s:8:"DISABLED";s:10:"dsChecksum";s:4:"none";s:12:"dsLogMessage";s:0:"";s:7:"content";a:2:{s:4:"type";s:6:"string";s:7:"content";s:523:"<?xml version="1.0" encoding="UTF-8"?> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:fedora="info:fedora/fedora-system:def/relations-external#" xmlns:fedora-model="info:fedora/fedora-system:def/model#" xmlns:islandora="http://islandora.ca/ontology/relsext#"> <rdf:Description rdf:about="info:fedora/npc:lanz"> <fedora:isMemberOfCollection rdf:resource="info:fedora/npc:root"/> <fedora-model:hasModel rdf:resource="info:fedora/islandora:collectionCModel"/> </rdf:Description> </rdf:RDF> ";}}}}s:13:"relationships";O:13:"FedoraRelsExt":6:{s:6:"*new";b:0;s:11:"*domCache";N;s:10:"datastream";r:203;s:13:"*namespaces";a:4:{s:3:"rdf";s:43:"http://www.w3.org/1999/02/22-rdf-syntax-ns#";s:6:"fedora";s:49:"info:fedora/fedora-system:def/relations-external#";s:12:"fedora-model";s:36:"info:fedora/fedora-system:def/model#";s:9:"islandora";s:37:"http://islandora.ca/ontology/relsext#";}s:6:"object";r:1;s:18:"nonMagicAutoCommit";b:1;}s:10:"repository";r:207;s:11:"*objectId";s:8:"npc:lanz";s:16:"*objectProfile";a:4:{s:8:"objState";s:1:"A";s:10:"objOwnerId";s:5:"admin";s:8:"objLabel";s:0:"";s:13:"objLogMessage";s:0:"";}s:9:"resources";a:0:{}} [:db_insert_placeholder_3] => 19 ) in IslandoraBatchPreprocessor->addToDatabase() (line 199 of /var/www/drupal/sites/all/modules/islandora_batch/includes/preprocessor_base.inc).
@bondjimbond hey. Sorry, morning of calls, teaching and stuff. So back to your issues
>>
in your logs are conflict messages of a badly/failed merge/badly patched. Checking out our branch should fix that. Checkout the branch.
PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'npc:lanz' for key 'PRIMARY': INSERT INTO {islandora_batch_queue} (id, parent, data, sid) ... >addToDatabase() (line 199 of /var/www/drupal/sites/all/modules/islandora_batch/includes/preprocessor_base.inc).
Sorry @DiegoPino, I must have been reporting logs from a different attempt.
This time, I have made made sure that:
The batch set is created successfully, but processing the set results in errors for each object: "Serious datastream issues on npc:absheppard have forced us to not ingest it."
There are no errors in the Drupal log. The Apache error log tells me nothing either:
[Tue May 26 17:59:30.625840 2020] [:error] [pid 2537] [client 10.0.2.2:56266] Running Multi Importer getIngestInfo, referer: http://localhost:8000/multi_importer
In case this helps..
Parameters for Set 20
Array
(
[type] => LOCAL
[source_data] => 62
[cmodel_source_field_index] => 2
[cmodel_mapping] => Array
(
[islandora:collectionCModel] => Array
(
[dsid] => Array
(
[rows] => Array
(
[DC] => Array
(
[method] => Array
(
[data] => xlst|0
)
)
[TN] => Array
(
[method] => Array
(
[data] =>
)
)
)
)
)
)
[object_maping] => Array
(
[pidmap_row] => Array
(
[pidmap] => 5
[pidtype] => 0
)
[parentmap_row] => Array
(
[parentmap] => 3
[parenttype] => 1
)
[labelmap_row] => Array
(
[labelmap] => 4
)
[sequencemap_row] => Array
(
[sequencemap] => 1
)
[dsmap_row] => Array
(
[dsremote] => LOCAL
)
[relsext_row] => Array
(
[relsext_mode] => NEW
)
)
[computed_cmodels] => Array
(
[islandora:collectionCModel] => Array
(
[DC] => Array
(
[id] => DC
[mime] => Array
(
[0] => application/xml
)
[optional] =>
)
[RELS-EXT] => Array
(
[id] => RELS-EXT
[mime] => Array
(
[0] => application/rdf+xml
)
[optional] => 1
)
[RELS-INT] => Array
(
[id] => RELS-INT
[mime] => Array
(
[0] => application/rdf+xml
)
[optional] => 1
)
[TN] => Array
(
[id] => TN
[mime] => Array
(
[0] => image/jpeg
[1] => image/png
[2] => image/gif
)
[optional] =>
)
)
)
[action] => ingest
)
Hi, where is MODS in that mapping?
Diego Pino Navarro Assistant Director for Digital Strategy Metropolitan New York Library Council 599 11th Av. New York, NY 10036
On May 26, 2020, at 2:13 PM, Brandon Weigel notifications@github.com wrote:
In case this helps..
Parameters for Set 20 Array ( [type] => LOCAL [source_data] => 62 [cmodel_source_field_index] => 2 [cmodel_mapping] => Array ( [islandora:collectionCModel] => Array ( [dsid] => Array ( [rows] => Array ( [DC] => Array ( [method] => Array ( [data] => xlst|0 )
) [TN] => Array ( [method] => Array ( [data] => ) ) ) ) ) ) [object_maping] => Array ( [pidmap_row] => Array ( [pidmap] => 5 [pidtype] => 0 ) [parentmap_row] => Array ( [parentmap] => 3 [parenttype] => 1 ) [labelmap_row] => Array ( [labelmap] => 4 ) [sequencemap_row] => Array ( [sequencemap] => 1 ) [dsmap_row] => Array ( [dsremote] => LOCAL ) [relsext_row] => Array ( [relsext_mode] => NEW ) ) [computed_cmodels] => Array ( [islandora:collectionCModel] => Array ( [DC] => Array ( [id] => DC [mime] => Array ( [0] => application/xml ) [optional] => ) [RELS-EXT] => Array ( [id] => RELS-EXT [mime] => Array ( [0] => application/rdf+xml ) [optional] => 1 ) [RELS-INT] => Array ( [id] => RELS-INT [mime] => Array ( [0] => application/rdf+xml ) [optional] => 1 ) [TN] => Array ( [id] => TN [mime] => Array ( [0] => image/jpeg [1] => image/png [2] => image/gif ) [optional] => ) ) ) [action] => ingest
) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/pull/117#issuecomment-634190860, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU7ZZ3KY3IMHKNP5GPCBPDRTQBFBANCNFSM4NIDQMQQ.
Ah, a good question! So for the Collection CModel mapping, MODS is not one of the options presented!
Did a quick check of another CSV with Video objects, and the MODS menu item is there.
So what we've identified here, then, is a bug in the Collection CModel mapping?
hahahaha. Yes. Not there. MODS not there! All my Servers had a modified CMODEL that includes MODS, i forget this this is another islandora bug..) https://github.com/Islandora/islandora_solution_pack_collection/blob/7.x/xml/islandora_basic_collection_ds_composite_model.xml https://github.com/Islandora/islandora_solution_pack_collection/blob/7.x/xml/islandora_basic_collection_ds_composite_model.xml Ok, so you have 3 options. 1.- Generate just DC via a twig template (not via XSLT) 2.- Modify the CMODEL to allow/enforce MODS or. 3.- Enable the add extra DS option in IMI and add, there in the CMODEL mapping, in realtime a MODS datastream that uses the twig template
Diego Pino Navarro Assistant Director for Digital Strategy Metropolitan New York Library Council 599 11th Av. New York, NY 10036
On May 26, 2020, at 2:50 PM, Brandon Weigel notifications@github.com wrote:
Ah, a good question! So for the Collection CModel mapping, MODS is not one of the options presented! https://user-images.githubusercontent.com/6069170/82938603-14a11800-9f60-11ea-846f-f4ff0ca0ca43.png Did a quick check of another CSV with Video objects, and the MODS menu item is there.
So what we've identified here, then, is a bug in the Collection CModel mapping?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/pull/117#issuecomment-634209945, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU7ZZ3XKXCVL476O7EVSADRTQFOXANCNFSM4NIDQMQQ.
Created a PR on the Collection SP to fix that. Can you review? This is the easiest solution, and is an overall improvement/bugfix which Islandora should have.
With the PR we're discussing and my PR on the Collection SP, it becomes possible to ingest collections with IMI.
It would be nice to expand the "Local" label to say "Local file or no files", so people know which to pick if they do have no files.
Thanks for merging that PR! Now for this one. :) Once "or no files" is added to that option label, I think this should be good to go.
@bondjimbond yes. Will deal with that tonight. Stuck with Drupal 9 checks and Archipelago and training also, 24 hours a days feels way too short. Thanks!
The label change looks good!
Are you ready to merge?
Merged. We need some testers now that break everything! Thanks!!
What is new?
See #116
This pull adds an option in IMI's admin config (admin/islandora/tools/multi_importer) that overrides our CMODEL sanity checking for required/optional datastreams and adds, if the new checkbox is enabled, an all over the place --Don't Create-- Option, except, if present, for MODS, when enforced (Hope this makes sense!)
This pull also adds a kinda better message in case the Google Token has expired and does not allow a grant again, which will happen if you test this pull by saving the config form and you had not make changes recently to your credentials (old old old).
Finally, most expected i guess, it makes some quite rough decisions for you/defaults when setting the CMODEL mapping. Now some Datastreams will show certain defaults instead of the first option in the settings, allowing you for the first time to click less and enjoy more.
@bondjimbond as promised, here it is and its still Friday! Please test this branch. We can still tune some options here, always in the context of course of that original ISSUE. Also, if you set everything to DON'T CREATE, ingest will not happen and you will get again the error
Serious datastream issues on @pid have forced us to not ingest it.
Which will always trigger if your OBJECT is 100% datastreamless, for whatever reason that is. I hardcoded that so. If you require empty Objects to be ingested, we need to talk about this