Performing enrichments #2

Hi Folks, I'm building off an older issue in another repo here - Performing enrichments 14 - https://github.com/DigitalNZ/supplejack_docker.

We've been working with the Supplejack Docker repo for ~12 months or so as a prototype for a Provincial Digital Library. Many of our parsers in this build contain enrichments.

We've noticed that none of these enrichments successfully port over to the non-docker dev build (that you have graciously provided a walkthrough at: http://digitalnz.github.io/supplejack/start/installation-walk-through-by-example.html)

These enrichments invariably fail in this new build with the message Failed with exception ActiveResource::Collection#each delegated to to_a.each, but to_a is nil: nil - http://eln-sj7.is.sfu.ca:3002/sidekiq/morgue

I'm wondering if there are some key differences the way enrichments are configured in https://github.com/DigitalNZ/supplejack_api vs https://github.com/DigitalNZ/supplejack_api_app, that could account for this. I see issue about Restful activity in the API (#116), but I'm not sure if it's related to any of this.

Any advice or pointers you could provide our way would be greatly appreciated.

I've included one of our failing enrichments for reference [it pulls a thumbnail link from an adjacent mets file]

class WaterlooUniversityLibrary < SupplejackCommon::Oai::Base

    base_url "https://uwspace.uwaterloo.ca/oai/request"

  namespaces mets: "http://www.loc.gov/METS/",
               dc: "http://purl.org/dc/elements/1.1/",
            xlink: "http://www.w3.org/TR/xlink/",
              dim: "http://www.dspace.org/xmlns/dspace/dim/"

  attribute :jurisdiction, default: "Ontario"
   #The jurisdiction field is identified in record_schema.rb. ,to allow us to filter results by region (and create API keys with roles assigned
  # to each jurisdiction. This allows us to create different front ends for different jurisdictions)

  attribute :internal_identifier, xpath: '//record/header/identifier'
  attribute :display_content_partner, default: 'University of Waterloo' 

  attribute :title, xpath: '//dc:title'
  attribute :description, xpath: '//dc:description'
  attribute :creator, xpath: '//dc:creator'
  attribute :subject, xpath: '//dc:subject', mappings: {
    'Harvested from' => ''

                          }

  attribute :display_date do fetch('//dc:date').truncate(4, "").select(:last) #changed to match 4 digit standard
    end

  attribute :category, xpath: '//dc:type', mappings: {'Doctoral Thesis' => 'Dissertation/thesis'}

  attribute :language, xpath: '//dc:language'
  attribute :rights, xpath: '//dc:rights'
  attribute :publisher, xpath: '//dc:publisher'

  attribute :source_url, xpath: '//dc:identifier' do
    get(:source_url).find_with('hdl.handle.net')
  end

  enrichment :get_thumbnail, priority: -4, required_for_active_record: false do

    requires :uri do
      primary[:source_url].mapping(/^.*handle\.net(.*)$/ => 'https://uwspace.uwaterloo.ca/metadata/handle\1/mets.xml').first
    end

    #for testing
    #attribute :subject, default: "#{requirements[:uri]}"

    url requirements[:uri]
    format :xml

  attribute :thumbnail_url, xpath: '//*[@USE="THUMBNAIL"]//@*[local-name()="href"]' do
    compose('https://uwspace.uwaterloo.ca', get(:thumbnail_url))
  end

  end
 end

Thanks for considering, Dan

Hi Dan,

Sorry about the late reply, we will be looking into this shortly for you and hope that we can resolve the issue.

Thanks, Richard

Hey @telezoic,

Thanks for reporting this issue, we have found the problem and fixed it in the most recent version of Supplejack Engine (v2.7.3). I have run the enrichment that you provided on my development machine and it has worked correctly.

Please update your engine to the most recent tag and let me know if the problem has been resolved.

Thanks, Richard

Thanks @Reinkaos!

I pulled and configured the api (and the manager and worker.yml). I also needed to add app/supplejack_api/record_schema.rb as it didn't seem to generate. I have all the pieces up, but all of my harvests (file/OAI/JSON) are failing with the following error message:

RestClient::NotFound
404 Not Found

/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:853:in `start'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient.rb:71:in `post'
/home/dsifton/supplejack/supplejack_worker/app/workers/api_update_worker.rb:31:in `perform'
/home/dsifton/supplejack/supplejack_worker/app/workers/harvest_worker.rb:84:in `post_to_api'
/home/dsifton/supplejack/supplejack_worker/app/workers/preview_worker.rb:91:in `enrich_record'
/home/dsifton/supplejack/supplejack_worker/app/workers/preview_worker.rb:35:in `block in perform'
/home/dsifton/supplejack/supplejack_worker/app/models/harvest_job.rb:61:in `block in records'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bundler/gems/supplejack_common-d7bf6f130c82/lib/supplejack_common/paginated_collection.rb:125:in `yield_from_records'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bundler/gems/supplejack_common-d7bf6f130c82/lib/supplejack_common/paginated_collection.rb:33:in `block in each'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bundler/gems/supplejack_common-d7bf6f130c82/lib/supplejack_common/paginated_collection.rb:30:in `each'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bundler/gems/supplejack_common-d7bf6f130c82/lib/supplejack_common/paginated_collection.rb:30:in `each'
/home/dsifton/supplejack/supplejack_worker/app/models/harvest_job.rb:60:in `each_with_index'
/home/dsifton/supplejack/supplejack_worker/app/models/harvest_job.rb:60:in `records'
/home/dsifton/supplejack/supplejack_worker/app/workers/preview_worker.rb:32:in `perform'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/processor.rb:151:in `execute_job'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/processor.rb:133:in `block (2 levels) in process'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/logging.rb:31:in `with_context'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/sidekiq-4.1.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'

I rebuild the whole stack after repeatedly bumping into this. But still, I get the same error.

I'm hoping you could shed some light on how I could correct this. Thanks again!

Hey @telezoic,

I would assume with a 404 error the host that you are trying to harvest might be down? Are you running all of the components of your stack?

Thanks, Richard

Hi @Reinkaos,

Sorry about the delay. At the time, all my endpoints were up and running (One of them was the DigitalNZ api :)) and as far as I could tell all the stack pieces. I'm hoping to do a fresh build next week and see how things go . . .

Hi @Reinkaos ,

I've come back to this with a fresh build.

I can confirm all the endpoints I'm trying to reach are available, and all my logging windows are up, each piece of the stack appears to be running, but I still see the following in my harvest preview window:

RestClient::NotFound
404 Not Found
    /home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
    /home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!...

But I can query directly from the RestClient:

irb(main):003:0> RestClient.get 'http://eln-sj4.is.sfu.ca/data/MHGLtemp.xml'
=> <RestClient::Response 200 "<?xml versi...">

the output at the api console gives :

actionpack (5.1.4) lib/action_dispatch/middleware/debug_exceptions.rb:63:in `call'
web-console (3.6.2) lib/web_console/middleware.rb:135:in `call_app'
web-console (3.6.2) lib/web_console/middleware.rb:30:in `block in call'
web-console (3.6.2) lib/web_console/middleware.rb:20:in `catch'
web-console (3.6.2) lib/web_console/middleware.rb:20:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
railties (5.1.4) lib/rails/rack/logger.rb:36:in `call_app'
railties (5.1.4) lib/rails/rack/logger.rb:24:in `block in call'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:69:in `block in tagged'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:69:in `tagged'
railties (5.1.4) lib/rails/rack/logger.rb:24:in `call'
sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:13:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/request_id.rb:25:in `call'
rack (2.0.5) lib/rack/method_override.rb:22:in `call'
rack (2.0.5) lib/rack/runtime.rb:22:in `call'
activesupport (5.1.4) lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/executor.rb:12:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/static.rb:125:in `call'
rack (2.0.5) lib/rack/sendfile.rb:111:in `call'
railties (5.1.4) lib/rails/engine.rb:522:in `call'
puma (3.12.0) lib/puma/configuration.rb:225:in `call'
puma (3.12.0) lib/puma/server.rb:658:in `handle_request'
puma (3.12.0) lib/puma/server.rb:472:in `process_client'
puma (3.12.0) lib/puma/server.rb:332:in `block in run'
puma (3.12.0) lib/puma/thread_pool.rb:133:in `block in spawn_thread'
Started POST "/harvester/records.json" for 127.0.0.1 at 2018-08-08 10:35:18 -0700

ActionController::RoutingError (No route matches [POST] "/harvester/records.json"):

The api url in a browser gives an action Controller error page that suggest adding routes to config/routes.rb. My knowledge of rails routing is lacking, so I tried the route from the supplejack docker build : mount SupplejackApi::Engine => '/', as: 'supplejack_api', but that just replaces the 404 preview errors with 403 errors. It fixes the error page though . . .

Adding the routes from https://github.com/DigitalNZ/supplejack_api/blob/master/config/routes.rb gives me loads of 'invalid route name' errors.

If you had any pointers, that would be terrific :).

Thanks and appreciation . . .

HI @telezoic. ActionController::RoutingError (No route matches [POST] "/harvester/records.json"): indicates it's your API that is 404'ing, not http://eln-sj4.is.sfu.ca/data/MHGLtemp.xml

So I'm guessing your api is trying to create the record, and it is failing.

Can you do the following:

in your api, run bundle exec rails routes | grep harvester, and provide the output
in your worker, get the HARVESTER_API_KEY, from your application.yml (example: AS4wGKGRVgEszC_iyEUc). Then, on the command line for your api:
- bundle exec rails c
- SupplejackApi::User.find_by_api_key('AS4wGKGRVgEszC_iyEUc').role, and confirm it is "harvester"

Also, can you provide the sha or tag number of the SupplejackApi gem engine you are using. The lastest is 7c9d04d1e5a9b02276a82451fd6f60ce59f53c81. The reason I'm checking is that the most recent commit changed the error codes :-)

Cheers Oliver

Thanks @ostigley!

I removed the older API and pulled latest ddd8ece869ee3800c9b27db9cfbc7196f209e400 [I've been off for a bit]

But, generating the harvester key via rails cin the api fails with

/home/dsifton/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/inflector/methods.rb:269:in `const_get': uninitialized constant RecordSchema (NameError)`

It looks like the schema wasn't created at app/supplejack_api/

I created a folder and added the schema - app/supplejack_api/record_schema.rb and it runs . . [included in the schemas is:]

# Roles
  role :harvester, default: true
  role :admin
  role :developer

end

then I proceed with bundle exec rails routes | grep harvester and after a few seconds nothing happens

[dsifton@eln-sj7 api]$ bundle exec rails routes | grep harvester
[dsifton@eln-sj7 api]$

[dsifton@eln-sj7 api]$ bundle exec rails routes
You don't have any routes defined!

Please add some routes in config/routes.rb.

and find_by_api_key

Loading development environment (Rails 5.1.4)
irb(main):001:0> SupplejackApi::User.find_by_api_key('MyKQ3A-zPX4A_jzJpZtg').role
MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
MONGODB | Topology type 'unknown' initializing.
MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=127.0.0.1:27017 topology=Unknown>
MONGODB | Server 127.0.0.1:27017 initializing.
MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
MONGODB | Topology type 'unknown' changed to type 'single'.
MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
MONGODB | Server description for 127.0.0.1:27017 changed from 'unknown' to 'standalone'.
MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
MONGODB | There was a change in the members of the 'single' topology.
MONGODB | 127.0.0.1:27017 | supplejack_api_development.find | STARTED | {"find"=>"users", "filter"=>{"authentication_token"=>"MyKQ3A-zPX4A_jzJpZtg"}, "sort"=>{"_id"=>1}, "limit"=>1, "singleBatch"=>true, "lsid"=>{"id"=><BSON::Binary:0x47412357392260 type=uuid data=0x712bbe17380a4e39...>}}
MONGODB | 127.0.0.1:27017 | supplejack_api_development.find | SUCCEEDED | 0.002s
=> "harvester"

I hope this is useful . . . and thanks again,

@telezoic we have a story in the backlog to come back to you over the next few days. Sorry for the problems you are experiencing!

sorry to be bringing you so many problems :)

From: Oliver Stigley notifications@github.com Sent: Thursday, August 30, 2018 1:32:39 PM To: DigitalNZ/supplejack_api Cc: Daniel Sifton; Mention Subject: Re: [DigitalNZ/supplejack_api] Performing enrichments #2 (#141)

@telezoichttps://github.com/telezoic we have a story in the backlog to come back to you over the next few days. Sorry for the problems you are experiencing!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DigitalNZ/supplejack_api/issues/141#issuecomment-417457631, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAbI6CpUFb0Aqxi7PPFLHTRS3eYVpe6pks5uWEvngaJpZM4U6QtL.

Hi @telezoic, my name is Theo, I work alongside Olly, Richard and Gus. Thanks again for submitting this issue! Appreciate you letting us know about these issues as we are always looking for ways we can improve Supplejack.

Just thought I'd let you know that we have identified an issue with the supplejack_api:install generator and are working on resolving https://github.com/DigitalNZ/supplejack_api/pull/153. There's a little write up on the issue. Essentially, the generator was trying to copy files that did not exist and would fizzle out. Will try get this into master ASAP

As well as that I am still taking a look at why you are getting 404s for the XML file on your server. Hopefully should not take too much to get a solution for you :)

Any further questions please feel free to reach out. Cheers!

Hey Daniel, just thought I'd let you know I've taken a look at the XML file you were trying to parse with SJ. Harvests are working fine with your data. Please let us know if you have not resolved your issue and if you have any further questions. Thanks :)

Awesome! It's all up and running! Thanks @theomarkkuspaul! I'm going to run a few more tests and start building for a 'production install' via http://digitalnz.github.io/supplejack/start/production-install.html

Great to hear you got around your problems! Any further questions please feel free to get in touch.

Ok. One more, I think. :) I can harvest, and enrich, which is great! Thanks again! But my API doesn't respond to queries. and seems to show too many fragments (my other APIs on different SJ installs don't look like this)

http://eln-sj7.is.sfu.ca:3000/v3/records.json?api_key=Rf5Avy1rk6ApyNLQ9YNB&per_page=5

I set per_page=5 as a sample here. It always shows the default 20 items, and any query gives the same results, the default 20.

I can, however, retrieve a single record: http://eln-sj7.is.sfu.ca:3000/v3/records/13702.json?api_key=Rf5Avy1rk6ApyNLQ9YNB

But I can't filter that further: http://eln-sj7.is.sfu.ca:3000/v3/records/13702.json?api_key=Rf5Avy1rk6ApyNLQ9YNB&fields=title

I here's the api log:

[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99eff9ebdbdfc3310604a7, created_at: 2018-09-13 05:04:57 UTC, updated_at: 2018-09-13 05:04:57 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/441/NQ44760.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effaebdbdfc3310604aa, created_at: 2018-09-13 05:04:58 UTC, updated_at: 2018-09-13 05:04:58 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/676/NQ65233.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effaebdbdfc3310604ad, created_at: 2018-09-13 05:04:58 UTC, updated_at: 2018-09-13 05:04:58 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/298/nq32817.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effaebdbdfc3310604b0, created_at: 2018-09-13 05:04:58 UTC, updated_at: 2018-09-13 05:04:58 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/672/NQ65229.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effaebdbdfc3310604b3, created_at: 2018-09-13 05:04:58 UTC, updated_at: 2018-09-13 05:04:58 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/624/NQ60543.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effbebdbdfc3310604b6, created_at: 2018-09-13 05:04:59 UTC, updated_at: 2018-09-13 05:04:59 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/507/NQ51216.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effbebdbdfc3310604b9, created_at: 2018-09-13 05:04:59 UTC, updated_at: 2018-09-13 05:04:59 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/649/NQ60568.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b99effbebdbdfc3310604bc, created_at: 2018-09-13 05:04:59 UTC, updated_at: 2018-09-13 05:04:59 UTC, source_id: "get_thumbnail", priority: -4, job_id: "5b99eff7ebdbdfae48a1153d", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: nil, status: nil, title: nil, description: nil, display_collection: nil, display_content_partner: nil, category: nil, thumbnail_url: "https://uwspace.uwaterloo.ca/bitstream/handle/10012/305/NQ32824.pdf.jpg?sequence=5&isAllowed=y", source_url: nil, landing_url: nil>
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d78, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/1/data", status: "active", title: "CORSICAN in Lachine Rapids", description: "Stereoview of the Richelieu and Ontario Navigation Company steamboat CORSICAN in the Lachine Rapids near Montreal.", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00001t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/1/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/1/data">
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d7a, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/8/data", status: "active", title: "The CHICORA on Lake Ontario", description: "Stereoview of the ex-Civil War blockade runner CHICORA which came to the Great Lakes in 1868. She operated on the upper lakes until being bought by the Niagara Navigat...", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00003t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/8/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/8/data">
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d7c, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/10/data", status: "active", title: "Beaumaris Lake, Muskoka", description: "Stereoview photograph of the MUSKOKA coming in the the wharf on Beaumaris Lake in the Muskokas", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00005t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/10/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/10/data">
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d7b, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/7/data", status: "active", title: "CORINTHIAN in rapids of the Saint Lawrence River", description: "Stereoview of the Richelieu and Ontario Navigation Company steamboat CORINTHIAN in the rapids of the Saint Lawrence River. Probably the Lachine Rapids.", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00002t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/7/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/7/data">
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d7e, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/9/data", status: "active", title: "Lachine Rapids", description: "A stereoview engraving of a steamboat running the Lachine Rapids above Montreal. It is probably intended to represent one of the Canadian Navigation Company steamboats...", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00004t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/9/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/9/data">
[active_model_serializers] No serializer found for resource: #<SupplejackApi::ApiRecord::RecordFragment _id: 5b9aa7bfebdbdfd998697d81, created_at: 2018-09-13 18:09:03 UTC, updated_at: 2018-09-13 18:09:03 UTC, source_id: "odw", priority: 0, job_id: "5b9aa7beebdbdfae047331b2", _type: "SupplejackApi::ApiRecord::RecordFragment", internal_identifier: "http://images.maritimehistoryofthegreatlakes.ca/11/data", status: "active", title: "Steamer ST. LAWRENCE. Thousand Islands Series", description: "Stereoview of the Thousand Island Steamboat Company's steamboat ST. LAWRENCE in the Thousand Islands", display_collection: nil, display_content_partner: ["Our Digital World"], category: ["image"], thumbnail_url: "http://images.MaritimeHistoryoftheGreatLakes.ca/Images/MHGL00006t.jpg", source_url: "http://images.maritimehistoryofthegreatlakes.ca/11/data", landing_url: "http://images.maritimehistoryofthegreatlakes.ca/11/data">
[active_model_serializers] Rendered ActiveModel::Serializer::CollectionSerializer with ActiveModelSerializers::Adapter::Attributes (69.82ms)
[active_model_serializers] Rendered SupplejackApi::SearchSerializer with ActiveModelSerializers::Adapter::Json (91.17ms)
MONGODB | 127.0.0.1:27017 | supplejack_api_development.insert | STARTED | {"insert"=>"record_interactions", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x70304319919940 type=uuid data=0x6af1ba99eb264c19...>}, "documents"=>[{"_id"=>BSON::ObjectId('5b9ab624ebdbdfd998697d94'), "request_type"=>"search", "log_values"=>[nil, ni...
MONGODB | 127.0.0.1:27017 | supplejack_api_development.insert | SUCCEEDED | 0.001s
MONGODB | 127.0.0.1:27017 | supplejack_api_development.insert | STARTED | {"insert"=>"supplejack_api_request_metrics", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x70304319919940 type=uuid data=0x6af1ba99eb264c19...>}, "documents"=>[{"_id"=>BSON::ObjectId('5b9ab624ebdbdfd998697d95'), "date"=>2018-09-13 00:00:00 UTC, "rec...
MONGODB | 127.0.0.1:27017 | supplejack_api_development.insert | SUCCEEDED | 0.001s

I'm using the default schema:


class RecordSchema
  include SupplejackApi::SupplejackSchema
  # Namespaces
  namespace :dc, url: 'http://purl.org/dc/elements/1.1/'
  # Fields
  string    :record_id, store: false
  string    :internal_identifier
  string    :title,  search_boost: 10,  search_as: [:filter, :fulltext], namespace: :dc
  string    :description,  search_boost: 2,  search_as: [:filter, :fulltext],  namespace:   :dc
  string    :display_collection,  search_as: [:filter, :fulltext],  namespace: :sj
  string    :display_content_partner, multi_value: true,  namespace: :dc
  string    :category,  multi_value: true,  search_as: [:filter]
  string    :thumbnail_url,  namespace: :sj

  string    :source_url
  string    :landing_url
  string    :status
  datetime  :created_at, date_format: '%y/%d/%m'

  # Groups
  group :default do
    fields [
      :record_id,
      :internal_identifier,
      :title,
      :description,
      :category,
      :display_content_partner,
      :display_collection,
      :source_url,
      :landing_url,
      :thumbnail_url
    ]
  end

  # Roles
  role :developer, default: true
  role :admin, admin: true
  role :harvester, harvester: true
end

Am I missing the serializer? Or maybe something else? I be grateful for any advice/suggestions you could provide.

Thanks again, D. :)

Hey @telezoic,

Thanks for submitting your issue, I will have a look into this now and see if I can find out what has happened.

Richard

Hey @telezoic,

I have found the cause for this bug, please use version 2.9.3 of supplejack_api and let me know how you get on.

Thanks, Richard

Thanks Richard! Everything is up and running, and looks great!

From: Richard Matthews [mailto:notifications@github.com] Sent: September-16-18 4:34 PM To: DigitalNZ/supplejack_api supplejack_api@noreply.github.com Cc: Daniel Sifton Daniel.Sifton@viu.ca; Mention mention@noreply.github.com Subject: Re: [DigitalNZ/supplejack_api] Performing enrichments #2 (#141)

Hey @telezoichttps://github.com/telezoic,

I have found the cause for this bug, please use version 2.9.3 of supplejack_api and let me know how you get on.

Thanks, Richard

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DigitalNZ/supplejack_api/issues/141#issuecomment-421853232, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAbI6Nk8bHnWXuHxxvNxQWj2X6bKNTckks5ubuAAgaJpZM4U6QtL.

DigitalNZ / supplejack_api

Performing enrichments #2 #141