Error message appeared in reindexing status for Slavic Collections in Dashboard

[kelea99]

Summary

The item numbers were off (not all items were in his exhibition that should be)for Thomas Keenan's Slavic Collections DPUL exhibit. I ran a reindex for the site and it is still re-populating. however, an error message is currently in the reindexing status (but the numbers are still growing, so the reindex is still happening). As long as the reindex works, Thomas will have what he needs by deadline. But we should probably look into why a manual reindex was needed and why an error message appeared in the reindexing status.

Impact

Please include hard deadlines, if the exhibit is part of an event, the issue is stopping work, etc. If the reindex works, Thomas will have what he needs. But we should check again before his deadline day.

Priority recommendation

• [ ] asap Only if reindex does not work
• [ ] within the next 3 weeks
• [ ] PO will prioritize

Sudden Priority Justification

Required if "asap" or "within the next 3 weeks" is checked. Add "Sudden Priority" and "Maintenance/Research labels ASAP to meet deadline for thomas, but only if the reindex does not work.

Expected behaviour

I expect all open items from Figgy Collections to available in their DPUL sites. If I have to manually reindex the site, I expect The reindex to occur without an error message

Actual behaviour

All open items from Figgy Collections were not available in Thomas's DPUL site. I manually reindexed the site and it threw an error message in the reindexing status within the dashboard. However, the numbers being reindexed continue to grow, inferring the reindex is still happening.

Steps to reproduce behaviour

1. Go to dasboard
2. go to Items
3. Select reindex
4. wait

Screenshots

[hackartisan]

The reindex got all the items in.

[hackartisan]

I found the error! If you go to the dashboard, in the list of "recent item indexing activity" click on "failed" and you'll get a modal with some formatting problems.

It looks like these errors get captured here instead of sending to honeybadger, and they also keep the jobs from retrying? Seems potentially related to #1294

[hackartisan]

Here's an example one (they all look pretty much like this):

undefined method `[]' for nil:NilClass

{
  "backtrace": "/opt/dpul/releases/20220328195027/app/services/iiif_manifest.rb:74:in `noid'\n/opt/dpul/releases/20220328195027/app/services/iiif_manifest.rb:18:in `add_noid'\n/opt/dpul/releases/20220328195027/app/services/iiif_manifest.rb:6:in `to_solr'\n/opt/dpul/shared/bundle/ruby/2.7.0/gems/blacklight-spotlight-3.0.3/app/models/spotlight/resources/iiif_harvester.rb:23:in `block (2 levels) in indexing_pipeline'\n/opt/dpul/shared/bundle/ruby/2.7.0/gems/blacklight-spotlight-3.0.3/app/services/spotlight/etl/step.rb:24:in `block (2 levels) in call'",
  "resource_id": null
}

[hackartisan]

Most of these are this same "Undefined method [] for nil" error. Some are timeouts and other things like that.

It doesn't seem possible to tie the error back to the figgy resource. The error has a bit of backtrace and then a job tracker id. There's a spot for the resource id but it's nil.

[hackartisan]

A reindex on staging went through with no errors. There were errors on prod. Staging doesn't seem to index campus-only materials, while prod does. Maybe there's a problem with those resources.

[hackartisan]

here is the resource that is erroring: https://figgy.princeton.edu/catalog/d2d488af-2980-4554-a9ef-aacbaf463ec8

[kelea99]

I know there was a bug with DPUL prod and a visibility setting that @tpendragon checked for me. I think there is a ticket. Looking.

Here it is: https://github.com/pulibrary/dpul/issues/1300

[hackartisan]

@kelea99 thank you!! It seems likely that is indeed the same issue.