Investigate why fitch images are not all showing

jcoyne commented 8 years ago

in this view: https://exhibits-stage.stanford.edu/map-view-testing/catalog?q=&search_field=search&utf8=%E2%9C%93&view=heatmaps

only one result is showing on the map, but we expect 3.

Are there missing srpt fields? Is this a problem with geonames username.

ggeisler commented 8 years ago

This ticket is still relevant, but note that none of the three Fitch items are showing up on the map view (the one item in California I thought was Fitch was in fact a different non-fitch item).

atz commented 8 years ago

OK, I see two results on that map. One around Africa, and the other around Stanford, CA.

Here's the items currently in collection:

[ ] https://purl.stanford.edu/mp281vy9386.mods -- Africa -- appears
[ ] https://purl.stanford.edu/rh234sw2751.mods
[ ] https://purl.stanford.edu/cw222pt0426.mods -- Stanford -- appears
[ ] https://purl.stanford.edu/fs041zx5160.mods
[ ] https://purl.stanford.edu/vx013gz7298.mods

Which three are you expecting to appear on the map? I'm surprised for another reason. This morning, the hypothesis is that we failed to configure a geonames username. Looking at shared_configs, this appears to be true. In testing that meant no geonames were resolved. Since geoname resolution is currently the only code wired in to produce SRPT fields, I now expect nothing to display.

Analysis per record, assuming we configure the username correctly:

mp281vy9386: no geoname, no extension -- should not appear
rh234sw2751: has geoname, should appear
cw222pt0426: has geoname (plus extension and cartographic coordinates) -- should appear
fs041zx5160: has geoname, should appear
vx013gz7298: has geoname, should appear

cbeer commented 8 years ago

Since geoname resolution is currently the only code wired in to produce SRPT fields

The old point_bbox field was also updated to point at the new field: https://github.com/sul-dlss/exhibits/blob/master/app/models/spotlight/dor/indexer.rb#L87

atz commented 8 years ago

Ah, that would explain how some data got in. That corresponds to:

Array(mods_ng_xml.subject.cartographics.coordinates)
  .map(&:text)
  .map { |n| Stanford::Mods::Coordinate.new(n) }
  .select(&:valid?))
  .map(&:as_envelope)
  .compact

Both appearing records have subject.cartographics.coordinates.

cbeer commented 8 years ago

https://github.com/sul-dlss/shared_configs/pull/96

atz commented 8 years ago

@cbeer merged your config changes. Currently pulling/copying them on stage-a, stage-b and worker-stage-a as documented in the README: https://github.com/sul-dlss/shared_configs/

atz commented 8 years ago

Looks like you already did it on prod.

atz commented 8 years ago

Reindex not really working 100% yet. gdor.log looks like:

I, [2016-08-05T15:04:08.883800 #105212]  INFO -- : indexing item cw222pt0426
I, [2016-08-05T15:04:11.827554 #105212]  INFO -- : Indexing item cw222pt0426 in resource 13 (0 / 0) (2971.0ms)
I, [2016-08-05T15:04:11.865185 #105212]  INFO -- : indexing item rh234sw2751
I, [2016-08-05T15:04:14.605128 #105212]  INFO -- : Indexing item rh234sw2751 in resource 13 (1 / 0) (2740.4ms)
I, [2016-08-05T15:04:14.638252 #105212]  INFO -- : indexing item mp281vy9386
I, [2016-08-05T15:04:17.953626 #105212]  INFO -- : Indexing item mp281vy9386 in resource 13 (2 / 0) (3316.3ms)
I, [2016-08-05T15:04:17.999319 #105212]  INFO -- : indexing item fs041zx5160
I, [2016-08-05T15:04:20.177133 #105212]  INFO -- : Indexing item fs041zx5160 in resource 13 (3 / 0) (2178.4ms)
I, [2016-08-05T15:04:20.236594 #105212]  INFO -- : indexing item vx013gz7298
I, [2016-08-05T15:04:22.514946 #105212]  INFO -- : Indexing item vx013gz7298 in resource 13 (4 / 0) (2278.8ms)
I, [2016-08-05T15:04:22.565226 #105212]  INFO -- : Indexing resource #<DorSolrDocumentBuilder:0x0000000ae3cf88 @resource=#<DorHarvester id: 13, exhibit_id: 12, type: "DorHarvester", url: nil, data: {"druid_list"=>"cw222pt0426\r\nrh234sw2751\r\nmp281vy9386\r\nfs041zx5160\r\nvx013gz7298", "collections"=>{}, "druid_info"=>{}}, indexed_at: "2016-08-05 22:04:03", created_at: "2016-08-04 15:05:33", updated_at: "2016-08-05 22:04:08", metadata: {"enqueued_at"=>"2016-08-05 22:04:01 UTC", "last_indexed_estimate"=>0, "last_indexed_count"=>5, "last_indexed_finished"=>nil, "last_index_elapsed_time"=>nil}, index_status: 0>> (est. 0 items) (14472.4ms)

Counts are off, but no errors evident. Might need to restart application for config change to take effect?

atz commented 8 years ago

Meanwhile production.log has only:

I, [2016-08-05T15:04:03.388749 #105212]  INFO -- : 2016-08-05T15:04:03-0700: [Worker(delayed_job.1 host:exhibits-worker-stage-a.stanford.edu pid:105212)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=139) RUNNING
I, [2016-08-05T15:04:03.646820 #105212]  INFO -- : [ActiveJob] [Spotlight::ReindexJob] [05582478-ff03-437a-8694-e3076af8b294] Performing Spotlight::ReindexJob from DelayedJob(default) with arguments: gid://exhibits/Spotlight::Exhibit/12
I, [2016-08-05T15:04:23.253371 #105212]  INFO -- : [ActiveJob] [Spotlight::ReindexJob] [05582478-ff03-437a-8694-e3076af8b294] Reindexing #<DorHarvester:0x0000000ae3d6b8> (batch size: 20) (19558.6ms)
I, [2016-08-05T15:04:23.253836 #105212]  INFO -- : [ActiveJob] [Spotlight::ReindexJob] [05582478-ff03-437a-8694-e3076af8b294] Performed Spotlight::ReindexJob from DelayedJob(default) in 19606.59ms
I, [2016-08-05T15:04:23.350820 #105212]  INFO -- : 2016-08-05T15:04:23-0700: [Worker(delayed_job.1 host:exhibits-worker-stage-a.stanford.edu pid:105212)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=139) COMPLETED after 19.9407
I, [2016-08-05T15:04:23.378542 #105212]  INFO -- : 2016-08-05T15:04:23-0700: [Worker(delayed_job.1 host:exhibits-worker-stage-a.stanford.edu pid:105212)] 1 jobs processed at 0.0498 j/s, 0 failed

atz commented 8 years ago

Redeployed master to stage, reindexed, still the same display results/state.

atz commented 8 years ago

We appear not to have released stanford-mods. Doing that now.

atz commented 8 years ago

stanford-mods released, gdor-indexer release is now the blocker. Don't have rubygems permissions for that:

https://github.com/sul-dlss/gdor-indexer/issues/78

atz commented 8 years ago

gdor-indexer released.

atz commented 8 years ago

Bumped up deps in exhibits, redeployed to stage, reindexed... and nothing changed.

atz commented 8 years ago

I have no idea what is going on. If I put the full mods record for one of the non-appearing records (rh234sw2751) into an integration test, it produces:

'geographic_srpt' => ['ENVELOPE(-119.9344,-119.655,36.91154,36.66216)']

How is that not happening on the worker during reindex? The record is being updated (as reflected by the solr timestamps) and contains the same mods: https://sul-solr.stanford.edu/solr/exhibits_stage/select?fl=*&indent=on&q=id:rh234sw2751&wt=json

No Faraday failure is logged, no other exceptions are noted, and no partial/mangled data is put in the field. I'm at a loss.

atz commented 8 years ago

@cbeer says:

user account not enabled to use the free webservice. Please enable it on your account page conveniently delivered without an http error. maybe we need some addl error handling on https://github.com/sul-dlss/exhibits/blob/master/app/models/spotlight/dor/indexer.rb#L116

Response was:

<geonames><status message="user account not enabled to use the free webservice. Please enable it on your account page: http://www.geonames.org/manageaccount " value="10" /></geonames>

So that is the sul account created rather than use @mejackreed's. Basically, the code and configuration is now working, but the remote account is configured to fail (without a proper HTTP code, of course). Assigning to Chris on the presumption he has the geonames login for the account.

atz commented 8 years ago

Chris reports he enabled the account and tested it successfully.

sul-dlss / exhibits

Investigate why fitch images are not all showing #307