search

mnylc / islandora_multi_importer

This is a flexible, twig based, all cmodel, tabular data to islandora Object importer with optional ZeroMQ processing
GNU General Public License v3.0
16 stars 15 forks source link

AJAX Error Checking CMODELs with Oral Histories #62

Closed McFateM closed 5 years ago

McFateM commented 6 years ago

I'm using the latest IMI code from 'master' coupled with ISLE and the Oral Histories content model. What was working a couple of months ago now throws the following AJAX error when I try to check CMODELs for a dataset that contains one or more Oral History records:

An AJAX HTTP error occurred.
HTTP Result Code: 500
Debugging information follows.
Path: /system/ajax
StatusText: Internal Server Error
ResponseText: Exception: Objects returned by FakeObject::getIterator() must be traversable or implement interface Iterator in islandora_oralhistories_islandora_oralhistoriesCModel_islandora_derivative() (line 304 of /var/www/html/sites/all/modules/contrib/islandora_solution_pack_oralhistories/islandora_oralhistories.module).

I need to learn how to run the debugger (via PHPStorm) in my local ISLE instance so I plan to try and look closely at the code now.

McFateM commented 6 years ago

I'm also going to post this as a potential issue in the Oral Histories SP repository too. Looks like it could just be unexpected CMODEL behavior in that body of code.

McFateM commented 6 years ago

False alarm. This is a duplicate of https://github.com/Islandora-Labs/islandora_solution_pack_oralhistories/issues/109, an issue that I address previously in the Oral Histories module but never created a PR for. Going to try to do that now.

MarcusBarnes commented 6 years ago

@McFateM Would you please create the pull-request so that the oral histories solution pack maintainers can incorporate your changes to address Islandora-Labs/islandora_solution_pack_oralhistories#109. It doesn't need to be perfect - just enough so that the maintainers can implement the changes that appear to help resolve this issue with multi-importer. If you feel uncomfortable making a pull-request, you could also send me the code changes via email and I'll review and make a pull-request. Thank you in advance!

wpwentzell commented 5 years ago

This issue is still persisting for us, using the latest versions of IMI and OHSP. This issue does not allow us to ingest any objects using multi-importer in batch, and basically has set us at a stand-still on this project. @McFateM, were you ever able to solve this issue? Mind reaching out to us directly so that we can get moving on our batch ingests?

@MarcusBarnes this would be a good time to tackle tidying up the IMI for OHSP, and also crafting some documentation for OHSP so that others who need to batch ingest can do so.

DiegoPino commented 5 years ago

To my understanding this is not an IMI problem and if so sadly there is nothing we can do on this side of the code. If you feel the problem is here i would encourage you to share your logs and any info regarding the problem so someone can reproduce the problem. Since OHSP is not maintaned by us (Diego or metro) we can only help by identifying the issue (which I don’t know which is right now given that orginal issue on the OHSP repo says resolved) and making a pull once we know what is wrong, basically same workflow you as user can follow to help move this forward. If you have info about where this fails please share it. Thanks

El El lun, 26 de ago. de 2019 a las 13:40, wpwentzell < notifications@github.com> escribió:

This issue is still persisting for us, using the latest versions of IMI and OHSP. This issue does not allow us to ingest any objects using multi-importer in batch, and basically has set us at a stand-still on this project. @McFateM https://github.com/McFateM, were you ever able to solve this issue? Mind reaching out to us directly so that we can get moving on our batch ingests?

@MarcusBarnes https://github.com/MarcusBarnes this would be a good time to tackle tidying up the IMI for OHSP, and also crafting some documentation for OHSP so that others who need to batch ingest can do so.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/issues/62?email_source=notifications&email_token=ABU7ZZ52UUWJWQDBJ4RSEGLQGQIRRA5CNFSM4EWWGB4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FC6JI#issuecomment-524955429, or mute the thread https://github.com/notifications/unsubscribe-auth/ABU7ZZZBETC63FO3TXDMTRLQGQIRRANCNFSM4EWWGB4A .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

McFateM commented 5 years ago

I have successfully used IMI with the OHSP since my last post here over a year ago, so I'm not sure what might have been wrong. It so happens that I have some more to ingest and I'm going to try that now just to see what happens. I'll report back, and if it works for me I'd be happy to share my configuration and assist if possible.

wpwentzell commented 5 years ago

@DiegoPino Thanks for your quick reply. I'll happily share details of our setup w/ logs. I'll collect that information and will return with the pertinent info.

@McFateM Thank you! You are the only person that we know to ever successfully ingest using IMI with OHSP. I have gone back over your code and pull requests from OHSP codebase and have done side by side comparison. We are using the latest release of OHSP (7.x-1.12) as well as the latest codebase for IMI. We are getting that exact AJAX error that you referenced in this issue ticket. Not sure how that is possible as, again, we are using the latest codebase that should have implemented Islandora-Labs/islandora_solution_pack_oralhistories#109.

wpwentzell commented 5 years ago

Just to confirm: we are using the latest OHSP release (7.x-1.12) and latest IMI codebase with Islandora ISLE. We get the exact AJAX error referenced above for this issue prior to it's closing:

Exception: Objects returned by FakeObject::getIterator() must be traversable or implement interface Iterator in islandora_oralhistories_islandora_oralhistoriesCModel_islandora_derivative() (line 304 of /var/www/html/sites/all/modules/islandora/islandora_solution_pack_oralhistories-7.x-1.12/islandora_oralhistories.module).

We have no trouble batch importing any other content model. We have tested extensively against the Large Image Solution Pack, for example, with no errors.

This error still seems to point to a fault in the OHSP codebase, as Diego notes.

Here are the relevant lines (299-308) referenced in the AJAX error: 


// Create a derivative if a text/vtt file is ingested and the config is true.
  if (variable_get('islandora_oralhistories_vtt_index_ds_generation', TRUE)) {
    // We don't know the datastream id,
    // thus need to filter if from the object keys.
    $arr_vtt_data_streams = array();
    foreach ($object as $key => $value) {
      if ($object[$key]->mimetype == "text/vtt") {
        array_push($arr_vtt_data_streams, $key);
      }
    }
wpwentzell commented 5 years ago

We are trying to ingest VTTs for transcripts. Our CSV is setup to have a column for transcripts, of which all are VTT files. Is it possible the derivative generation options within the OHSP are to fault for this?

McFateM commented 5 years ago

I just successfully, I think, ingested 11 new OH objects using IMI, but we don’t do VTT. Our transcripts are all in the native XML format. I’m happy to share our config and data structures if you think it might help, but if the issue is with VTT then I don’t think our setup will be much help.

From: wpwentzell notifications@github.com<mailto:notifications@github.com> Reply-To: mnylc/islandora_multi_importer reply@reply.github.com<mailto:reply@reply.github.com> Date: Monday, August 26, 2019 at 2:08 PM To: mnylc/islandora_multi_importer islandora_multi_importer@noreply.github.com<mailto:islandora_multi_importer@noreply.github.com> Cc: Mark McFate mcfatem@grinnell.edu<mailto:mcfatem@grinnell.edu>, Mention mention@noreply.github.com<mailto:mention@noreply.github.com> Subject: Re: [mnylc/islandora_multi_importer] AJAX Error Checking CMODELs with Oral Histories (#62)

We are trying to ingest VTTs for transcripts. Our CSV is setup to have a column for transcripts, of which all are VTT files. Is it possible the derivative generation options within the OHSP are to fault for this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mnylc_islandora-5Fmulti-5Fimporter_issues_62-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DACAURQKHAF3N5R75FJVTHIDQGQSZTA5CNFSM4EWWGB4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FLIRA-23issuecomment-2D524989508&d=DwMCaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=iq64txj0ssz2WNrbSHdmjen3fiCWXRj4e0H_DLqGmSA&s=3kDsK9it8LdPLvcOYvQNWaucO04ARL9fVo_YFO3pxQU&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACAURQPD2ZRO5X4WH65RQJ3QGQSZTANCNFSM4EWWGB4A&d=DwMCaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=iq64txj0ssz2WNrbSHdmjen3fiCWXRj4e0H_DLqGmSA&s=ulUuQlzqBZMFY9H4lkSO8pwU3i2-TNCCQ6mlsBAIYbs&e=.

wpwentzell commented 5 years ago

@McFateM That was it! If we turn off the option to Generate datastream for WebVTT indexing within the OHSP configuration, the CMODEL is parsed and is allowed to ingest.

So, indeed, there is a conflict in the OHSP codebase that prevents IMI ingest if also needing INDEXTRANSCRIPT datastream generation for solr. This makes sense considering the code referenced in the AJAX error we encountered. Simply deselecting this datastream generation fixes the issue. I will create a ticket for OHSP and reference my findings.

One step forward, and one step backward! Thanks for your help on this.

DiegoPino commented 5 years ago

Hi, i think i can come up with a fix/workaround in the few next days. Will research and let you all know here

Thanks to both of you for your feedback

El El lun, 26 de ago. de 2019 a las 15:41, wpwentzell < notifications@github.com> escribió:

@McFateM https://github.com/McFateM That was it! If we turn off the option to Generate datastream for WebVTT indexing within the OHSP configuration, the CMODEL is parsed and is allowed to ingest.

So, indeed, there is a conflict in the OHSP codebase that prevents IMI ingest if also needing INDEXTRANSCRIPT datastream generation for solr. This makes sense considering the code referenced in the AJAX error we encountered. Simply deselecting this datastream generation fixes the issue. I will create a ticket for OHSP and reference my findings.

One step forward, and one step backward! Thanks for your help on this.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/issues/62?email_source=notifications&email_token=ABU7ZZYTPV2LBQZJWAE4GJTQGQWV7A5CNFSM4EWWGB4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FOGEY#issuecomment-525001491, or mute the thread https://github.com/notifications/unsubscribe-auth/ABU7ZZ4GQICUZREQDHQJBYDQGQWV7ANCNFSM4EWWGB4A .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

DiegoPino commented 5 years ago

@McFateM @wpwentzell is just added a pull and a new branch (#105). If you can test and this works we can move to merge into master. Hope this helps.

wpwentzell commented 5 years ago

It works great, Diego! The INDEXTRANSCRIPT derivative fires off as expected. I'd say this is a complete success. The AJAX error is avoided entirely.

DiegoPino commented 5 years ago

Will give this a few more hours in case someone wants to comment or has some concerns. If not, this will be merged into master by the end of the day. Thanks for reporting the issue and pinpointing to the offending code, that made fixing this much easier!

DiegoPino commented 5 years ago

Resolved via https://github.com/mnylc/islandora_multi_importer/commit/8cb4eceb09f80d2b685c59e6734ee57840d561f1