Open telezoic opened 5 years ago
Hi @telezoic
Would you be able to share the following files?
app/supplejack_api/record_schema.rb
Gemfile
Gemfile.lock
Feel free to email me if you don't want to share your files here.
Thanks @motizuki! No trouble.
class RecordSchema
include SupplejackApi::SupplejackSchema
# Namespaces
namespace :dc, url: 'http://purl.org/dc/elements/1.1/'
# Fields
string :record_id, store: false
string :internal_identifier
string :title, search_boost: 10, search_as: [:filter, :fulltext], namespace: :dc
string :description, search_boost: 2, search_as: [:filter, :fulltext], namespace: :dc
string :display_collection, search_as: [:filter, :fulltext], namespace: :sj
string :display_content_partner, multi_value: true, namespace: :dc
string :category, multi_value: true, search_as: [:filter]
string :source_url
string :landing_url
string :status
datetime :created_at, date_format: '%y/%d/%m'
# Groups
group :default do
fields [
:record_id,
:internal_identifier,
:title,
:description,
:category,
:display_content_partner,
:display_collection,
:source_url,
:landing_url
]
end
# Roles
role :developer, default: true
role :admin, admin: true
role :harvester, harvester: true
end
git_source(:github) do |repo_name|
repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
"https://github.com/#{repo_name}.git"
end
gem 'supplejack_api', git: 'http://github.com/DigitalNZ/supplejack_api.git'
gem 'sunspot_rails', '~> 2.2.0'
gem 'active_model_serializers', '~> 0.10.7'
gem 'mongoid_auto_increment'
gem 'whenever', '~> 0.10.0'
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.1.4'
# Use sqlite3 as the database for Active Record
gem 'sqlite3','1.3.13'
# Use Puma as the app server
gem 'puma', '~> 3.7'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# See https://github.com/rails/execjs#readme for more supported runtimes
# gem 'therubyracer', platforms: :ruby
# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 3.0'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'
# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development
group :development, :test do
# Call 'byebug' anywhere in the code to stop execution and get a debugger console
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
# Adds support for Capybara system testing and selenium driver
gem 'capybara', '~> 2.13'
gem 'selenium-webdriver'
end
group :development do
# Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
gem 'web-console', '>= 3.3.0'
gem 'listen', '>= 3.0.5', '< 3.2'
# Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
gem 'spring'
gem 'spring-watcher-listen', '~> 2.0.0'
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]
GIT
remote: http://github.com/DigitalNZ/supplejack_api.git
revision: c262ce8a72d7d91bceaca09a2a9cb15e47fdf270
specs:
supplejack_api (1)
active_model_serializers
activemodel-serializers-xml
activeresource
coffee-rails
compass-rails
dalli
devise (= 4.4.0)
devise-token_authenticatable (<= 1.0.0)
dimensions
dry-struct
dry-validation (= 0.13.0)
figaro
gabba
jquery-rails
json-ld
kaminari
kaminari-mongoid
lazy_high_charts
mimemagic
mongoid (= 7.0.1)
mongoid_auto_increment
paperclip
progressbar
rails (= 5.1.4)
responders
rest-client
rufus-scheduler
sass-rails
sidekiq
state_machine
sunspot_rails
sunspot_solr
thin
uglifier
unicode_utils
GEM
remote: https://rubygems.org/
specs:
actioncable (5.1.4)
actionpack (= 5.1.4)
nio4r (~> 2.0)
websocket-driver (~> 0.6.1)
actionmailer (5.1.4)
actionpack (= 5.1.4)
actionview (= 5.1.4)
activejob (= 5.1.4)
mail (~> 2.5, >= 2.5.4)
rails-dom-testing (~> 2.0)
actionpack (5.1.4)
actionview (= 5.1.4)
activesupport (= 5.1.4)
rack (~> 2.0)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.0.2)
actionview (5.1.4)
activesupport (= 5.1.4)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.0.3)
active_model_serializers (0.10.9)
actionpack (>= 4.1, < 6)
activemodel (>= 4.1, < 6)
case_transform (>= 0.2)
jsonapi-renderer (>= 0.1.1.beta1, < 0.3)
activejob (5.1.4)
activesupport (= 5.1.4)
globalid (>= 0.3.6)
activemodel (5.1.4)
activesupport (= 5.1.4)
activemodel-serializers-xml (1.0.2)
activemodel (> 5.x)
activesupport (> 5.x)
builder (~> 3.1)
activerecord (5.1.4)
activemodel (= 5.1.4)
activesupport (= 5.1.4)
arel (~> 8.0)
activeresource (5.1.0)
activemodel (>= 5.0, < 7)
activemodel-serializers-xml (~> 1.0)
activesupport (>= 5.0, < 7)
activesupport (5.1.4)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (~> 0.7)
minitest (~> 5.1)
tzinfo (~> 1.1)
addressable (2.6.0)
public_suffix (>= 2.0.2, < 4.0)
arel (8.0.0)
bcrypt (3.1.13)
bindex (0.7.0)
bson (4.5.0)
builder (3.2.3)
byebug (11.0.1)
capybara (2.18.0)
addressable
mini_mime (>= 0.1.3)
nokogiri (>= 1.3.3)
rack (>= 1.0.0)
rack-test (>= 0.5.4)
xpath (>= 2.0, < 4.0)
case_transform (0.2)
activesupport
childprocess (1.0.1)
rake (< 13.0)
chronic (0.10.2)
chunky_png (1.3.11)
climate_control (0.2.0)
coffee-rails (4.2.2)
coffee-script (>= 2.2.0)
railties (>= 4.0.0)
coffee-script (2.4.1)
coffee-script-source
execjs
coffee-script-source (1.12.2)
compass (0.12.2)
chunky_png (~> 1.2)
fssm (>= 0.2.7)
sass (~> 3.1)
compass-rails (2.0.0)
compass (>= 0.12.2)
concurrent-ruby (1.1.5)
connection_pool (2.2.2)
crass (1.0.4)
daemons (1.3.1)
dalli (2.7.10)
devise (4.4.0)
bcrypt (~> 3.0)
orm_adapter (~> 0.1)
railties (>= 4.1.0, < 5.2)
responders
warden (~> 1.2.3)
devise-token_authenticatable (1.0.0)
devise (= 4.4.0)
dimensions (1.3.0)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
dry-configurable (0.8.2)
concurrent-ruby (~> 1.0)
dry-core (~> 0.4, >= 0.4.7)
dry-container (0.7.0)
concurrent-ruby (~> 1.0)
dry-configurable (~> 0.1, >= 0.1.3)
dry-core (0.4.7)
concurrent-ruby (~> 1.0)
dry-equalizer (0.2.2)
dry-inflector (0.1.2)
dry-logic (0.6.1)
concurrent-ruby (~> 1.0)
dry-core (~> 0.2)
dry-equalizer (~> 0.2)
dry-struct (0.7.0)
dry-core (~> 0.4, >= 0.4.3)
dry-equalizer (~> 0.2)
dry-types (~> 0.15)
ice_nine (~> 0.11)
dry-types (0.15.0)
concurrent-ruby (~> 1.0)
dry-container (~> 0.3)
dry-core (~> 0.4, >= 0.4.4)
dry-equalizer (~> 0.2, >= 0.2.2)
dry-inflector (~> 0.1, >= 0.1.2)
dry-logic (~> 0.5, >= 0.5)
dry-validation (0.13.0)
concurrent-ruby (~> 1.0)
dry-configurable (~> 0.1, >= 0.1.3)
dry-core (~> 0.2, >= 0.2.1)
dry-equalizer (~> 0.2)
dry-logic (~> 0.5, >= 0.5.0)
dry-types (~> 0.14, >= 0.14)
erubi (1.8.0)
et-orbi (1.2.1)
tzinfo
eventmachine (1.2.7)
execjs (2.7.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
ffi (1.11.1)
figaro (1.1.1)
thor (~> 0.14)
fssm (0.2.10)
fugit (1.2.1)
et-orbi (~> 1.1, >= 1.1.8)
raabro (~> 1.1)
gabba (1.0.1)
globalid (0.4.2)
activesupport (>= 4.2.0)
hamster (3.0.0)
concurrent-ruby (~> 1.0)
hash-deep-merge (0.1.1)
http-cookie (1.0.3)
domain_name (~> 0.5)
i18n (0.9.5)
concurrent-ruby (~> 1.0)
ice_nine (0.11.2)
jbuilder (2.9.1)
activesupport (>= 4.2.0)
jquery-rails (4.3.3)
rails-dom-testing (>= 1, < 3)
railties (>= 4.2.0)
thor (>= 0.14, < 2.0)
json-ld (3.0.2)
multi_json (~> 1.12)
rdf (>= 2.2.8, < 4.0)
jsonapi-renderer (0.2.0)
kaminari (1.1.1)
activesupport (>= 4.1.0)
kaminari-actionview (= 1.1.1)
kaminari-activerecord (= 1.1.1)
kaminari-core (= 1.1.1)
kaminari-actionview (1.1.1)
actionview
kaminari-core (= 1.1.1)
kaminari-activerecord (1.1.1)
activerecord
kaminari-core (= 1.1.1)
kaminari-core (1.1.1)
kaminari-mongoid (1.0.1)
kaminari-core (~> 1.0)
mongoid
lazy_high_charts (1.5.8)
hash-deep-merge
link_header (0.0.8)
listen (3.1.5)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
ruby_dep (~> 1.2)
loofah (2.2.3)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
mini_mime (>= 0.1.1)
method_source (0.9.2)
mime-types (3.2.2)
mime-types-data (~> 3.2015)
mime-types-data (3.2019.0331)
mimemagic (0.3.3)
mini_mime (1.0.1)
mini_portile2 (2.4.0)
minitest (5.11.3)
mongo (2.8.0)
bson (>= 4.4.2, < 5.0.0)
mongoid (7.0.1)
activemodel (>= 5.1, < 6.0.0)
mongo (>= 2.5.1, < 3.0.0)
mongoid_auto_increment (0.1.5)
mongoid (>= 2.0)
multi_json (1.13.1)
multipart-post (2.1.1)
netrc (0.11.0)
nio4r (2.3.1)
nokogiri (1.10.3)
mini_portile2 (~> 2.4.0)
orm_adapter (0.5.0)
paperclip (6.1.0)
activemodel (>= 4.2.0)
activesupport (>= 4.2.0)
mime-types
mimemagic (~> 0.3.0)
terrapin (~> 0.6.0)
pr_geohash (1.0.0)
progressbar (1.10.0)
public_suffix (3.0.3)
puma (3.12.1)
raabro (1.1.6)
rack (2.0.7)
rack-protection (2.0.5)
rack
rack-test (1.1.0)
rack (>= 1.0, < 3)
rails (5.1.4)
actioncable (= 5.1.4)
actionmailer (= 5.1.4)
actionpack (= 5.1.4)
actionview (= 5.1.4)
activejob (= 5.1.4)
activemodel (= 5.1.4)
activerecord (= 5.1.4)
activesupport (= 5.1.4)
bundler (>= 1.3.0)
railties (= 5.1.4)
sprockets-rails (>= 2.0.0)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.0.4)
loofah (~> 2.2, >= 2.2.2)
railties (5.1.4)
actionpack (= 5.1.4)
activesupport (= 5.1.4)
method_source
rake (>= 0.8.7)
thor (>= 0.18.1, < 2.0)
rake (12.3.2)
rb-fsevent (0.10.3)
rb-inotify (0.10.0)
ffi (~> 1.0)
rdf (3.0.12)
hamster (~> 3.0)
link_header (~> 0.0, >= 0.0.8)
redis (4.1.1)
responders (2.4.1)
actionpack (>= 4.2.0, < 6.0)
railties (>= 4.2.0, < 6.0)
rest-client (2.0.2)
http-cookie (>= 1.0.2, < 2.0)
mime-types (>= 1.16, < 4.0)
netrc (~> 0.8)
rsolr (2.2.1)
builder (>= 2.1.2)
faraday (>= 0.9.0)
ruby_dep (1.5.0)
rubyzip (1.2.3)
rufus-scheduler (3.6.0)
fugit (~> 1.1, >= 1.1.6)
sass (3.7.4)
sass-listen (~> 4.0.0)
sass-listen (4.0.0)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
sass-rails (5.0.7)
railties (>= 4.0.0, < 6)
sass (~> 3.1)
sprockets (>= 2.8, < 4.0)
sprockets-rails (>= 2.0, < 4.0)
tilt (>= 1.1, < 3)
selenium-webdriver (3.142.3)
childprocess (>= 0.5, < 2.0)
rubyzip (~> 1.2, >= 1.2.2)
sidekiq (5.2.7)
connection_pool (~> 2.2, >= 2.2.2)
rack (>= 1.5.0)
rack-protection (>= 1.5.0)
redis (>= 3.3.5, < 5)
spring (2.0.2)
activesupport (>= 4.2)
spring-watcher-listen (2.0.1)
listen (>= 2.7, < 4.0)
spring (>= 1.2, < 3.0)
sprockets (3.7.2)
concurrent-ruby (~> 1.0)
rack (> 1, < 3)
sprockets-rails (3.2.1)
actionpack (>= 4.0)
activesupport (>= 4.0)
sprockets (>= 3.0.0)
sqlite3 (1.3.13)
state_machine (1.2.0)
sunspot (2.2.8)
pr_geohash (~> 1.0)
rsolr (>= 1.1.1, < 3)
sunspot_rails (2.2.8)
rails (>= 3)
sunspot (= 2.2.8)
sunspot_solr (2.3.0)
terrapin (0.6.0)
climate_control (>= 0.0.3, < 1.0)
thin (1.7.2)
daemons (~> 1.0, >= 1.0.9)
eventmachine (~> 1.0, >= 1.0.4)
rack (>= 1, < 3)
thor (0.20.3)
thread_safe (0.3.6)
tilt (2.0.9)
turbolinks (5.2.0)
turbolinks-source (~> 5.2)
turbolinks-source (5.2.0)
tzinfo (1.2.5)
thread_safe (~> 0.1)
uglifier (4.1.20)
execjs (>= 0.3.0, < 3)
unf (0.1.4)
unf_ext
unf_ext (0.0.7.6)
unicode_utils (1.4.0)
warden (1.2.8)
rack (>= 2.0.6)
web-console (3.7.0)
actionview (>= 5.0)
activemodel (>= 5.0)
bindex (>= 0.4.0)
railties (>= 5.0)
websocket-driver (0.6.5)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.3)
whenever (0.10.0)
chronic (>= 0.6.3)
xpath (3.2.0)
nokogiri (~> 1.8)
PLATFORMS
ruby
DEPENDENCIES
active_model_serializers (~> 0.10.7)
byebug
capybara (~> 2.13)
coffee-rails (~> 4.2)
jbuilder (~> 2.5)
listen (>= 3.0.5, < 3.2)
mongoid_auto_increment
puma (~> 3.7)
rails (~> 5.1.4)
sass-rails (~> 5.0)
selenium-webdriver
spring
spring-watcher-listen (~> 2.0.0)
sqlite3 (= 1.3.13)
sunspot_rails (~> 2.2.0)
supplejack_api!
turbolinks (~> 5)
tzinfo-data
uglifier (>= 1.3.0)
web-console (>= 3.3.0)
whenever (~> 0.10.0)
BUNDLED WITH
2.0.1
@telezoic
Can you try adding this code into your record schema?
group :valid_set_fields do
includes []
fields []
end
Hi @telezoic, have you had a chance to try the suggested fix above?
Hi @motizuki,
Sorry, I've been away for a few weeks, and have just come back at this. I modified the schema as requested, and then restarted the API with:
bundle exec rails s -e production -b 0.0.0.0 -p 3000
This gives me a new 'nilclass' error, this time at:
...
3: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:3:in `<top (required)>'
2: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:4:in `<module:SupplejackApi>'
1: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:28:in `<class:UserSetSerializer>'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:29:in `<class:SetItemSerializer>': undefined method `fields' for nil:NilClass (NoMethodError)
Thanks for any advice you could share.
Hi @motizuki,
I found a way out of this error [above]:
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:29:in `<class:SetItemSerializer>': undefined method `fields' for nil:NilClass (NoMethodError)
Looking at your 'record_schema.rb` edit for the earlier error, I added another:
group :sets do
includes []
fields []
end
I have all the apps running production now . . . with one error in the manager that I'm attempting to troubleshoot.
Regardless of the target or parser, my harvests fail with the following message in the preview window:
ActiveResource::ResourceNotFound
Failed. Response code = 404. Response message = Not Found.
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:154:in `handle_response'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:133:in `request'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:92:in `block in get'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:227:in `with_auth'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:92:in `get'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/base.rb:1051:in `find_single'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/base.rb:938:in `find'
/data/sites/supplejack_worker/app/models/abstract_job.rb:82:in `parser'
/data/sites/supplejack_worker/app/models/harvest_job.rb:56:in `records'
/data/sites/supplejack_worker/app/workers/preview_worker.rb:34:in `perform'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:185:in `execute_job'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:167:in `block (2 levels) in process'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:166:in `block in process'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:108:in `local'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:42:in `block in call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/reloader.rb:73:in `block in wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/reloader.rb:72:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:41:in `call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:243:in `stats'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_logger.rb:8:in `call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:73:in `global'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:125:in `block in dispatch'
I can curl my harvest targets from my server and see that they are indeed up and running.
The manager log doesn't help me much :). A sample below
method=POST path=/parsers/5d546d9b2a041875cb1e165c/previewers format=*/* controller=PreviewersController action=create status=200 duration=108.45 view=2.03 params={"parser"=>{"content"=>"class SJHorses < SupplejackCommon::Json::Base\r\n\r\n \r\n #a horse specific search against the digitalNZ API\r\n\r\n\tbase_url \"http://api.digitalnz.org/v3/records.json?api_key=[]&fields=title,display_content_partner,creator,landing_url,source_url,thumbnail_url,subject,rights,category,description,id,date,publisher,language,locations&geo_bbox=-41,174,-42,175&text=horse\"\r\n \r\n\r\n record_selector \"$.search.results\"\r\n\r\n attribute :display_content_partner, default: \"South Pacific Surprise!\"\r\n attribute :category, default: \"image\"\r\n \r\n attribute :publisher, path: \"$.publisher\"\r\n attribute :subject, path: \"$.subject\"\r\n attribute :language, path: \"$.language\"\r\n attribute :title, path: \"$.title\"\r\n attribute :description, path: \"$.description\", truncate: 170\r\n attribute :source_url, path: \"$.source_url\"\r\n attribute :landing_url, path: \"$.landing_url\"\r\n attribute :thumbnail_url, path: \"$.thumbnail_url\"\r\n attribute :internal_identifier do get(:landing_url).downcase\r\n\tend\r\n attribute :display_date, path: \"$.date\", truncate: {length: 4, omission: \"\"}\r\n attribute :locations, path: \"$.locations\"\r\n \r\n \r\nend"}, "environment"=>"staging", "index"=>"1", "parser_id"=>"5d546d9b2a041875cb1e165c"} time=2019-08-14 21:24:14 +0000
method=GET path=/previews/5d547bff2a04187fc31e917c.js format=js controller=PreviewsController action=show status=200 duration=51.16 view=1.50 params={} time=2019-08-14 21:24:17 +0000
Here's a snippet of the worker log where I'm viewing records in the manager preview window:
D, [2019-08-14T21:27:12.517745 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.find | SUCCEEDED | 0.011s
D, [2019-08-14T21:27:12.519066 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"active", "updated_at"=>...
D, [2019-08-14T21:27:12.519782 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.521364 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"start_time"=>2019-08-14 21:27:12 ...
D, [2019-08-14T21:27:12.522120 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.774962 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"harvest_failure"=>{"_id"=>BSON::O...
D, [2019-08-14T21:27:12.775753 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.776854 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status_message"=>"Failed. Respon...
D, [2019-08-14T21:27:12.779217 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.002s
D, [2019-08-14T21:27:12.780880 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"failed", "updated_at"=>...
D, [2019-08-14T21:27:12.790886 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.010s
D, [2019-08-14T21:27:12.792796 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"end_time"=>2019-08-14 21:27:12 UT...
D, [2019-08-14T21:27:12.793608 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.795137 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"finished", "updated_at"...
D, [2019-08-14T21:27:12.795858 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.797444 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"end_time"=>2019-08-14 21:27:12 UT...
D, [2019-08-14T21:27:12.797982 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.000s
D, [2019-08-14T21:27:12.799627 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"previews", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e9180')}, "u"=>{"$set"=>{"harvest_failure"=>"{\"_id\":\"5d547cb0...
D, [2019-08-14T21:27:12.800423 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
I, [2019-08-14T21:27:13.873749 #32707] INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Started GET "/previews/5d547cb02a04187fc31e9180.json" for 127.0.0.1 at 2019-08-14 21:27:13 +0000
I, [2019-08-14T21:27:13.874323 #32707] INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Processing by PreviewsController#show as JSON
I, [2019-08-14T21:27:13.874371 #32707] INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Parameters: {"id"=>"5d547cb02a04187fc31e9180"}
D, [2019-08-14T21:27:13.875347 #32707] DEBUG -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] MONGODB | localhost:27017 | worker_production.find | STARTED | {"find"=>"previews", "filter"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e9180')}, "$readPreference"=>{"mode"=>"primary"}, "lsid"=>{"id"=><BSON::Binary:0x47211718073360 type=uuid data=0xebb98896b03849b1...>}}
D, [2019-08-14T21:27:13.885652 #32707] DEBUG -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] MONGODB | localhost:27017 | worker_production.find | SUCCEEDED | 0.010s
I, [2019-08-14T21:27:13.886763 #32707] INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] [active_model_serializers] Rendered ActiveModel::Serializer::Null with Preview (0.57ms)
I, [2019-08-14T21:27:13.886910 #32707] INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Completed 200 OK in 12ms (Views: 0.9ms)
Here's a parser I'm using:
[I've removed the api_key from the post]
class SouthPacificSurprise < SupplejackCommon::Json::Base
#a horse specific search against the digital NZ API
base_url "http://api.digitalnz.org/v3/records.json?api_key=[]&fields=title,display_content_partner,creator,landing_url,source_url,thumbnail_url,subject,rights,category,description,id,date,publisher,language,locations&geo_bbox=-41,174,-42,175&text=horse"
record_selector "$.search.results"
attribute :display_content_partner, default: "South Pacific Surprise!"
attribute :category, default: "image"
attribute :publisher, path: "$.publisher"
attribute :subject, path: "$.subject"
attribute :language, path: "$.language"
attribute :title, path: "$.title"
attribute :description, path: "$.description", truncate: 170
attribute :source_url, path: "$.source_url"
attribute :landing_url, path: "$.landing_url"
attribute :thumbnail_url, path: "$.thumbnail_url"
attribute :internal_identifier do get(:landing_url).downcase
end
attribute :display_date, path: "$.date", truncate: {length: 4, omission: ""}
attribute :locations, path: "$.locations"
end
Can you recommend another log that might be helpful here, or any other advice that might set me on the right path? :)
Thanks,
D
Hey @telezoic,
Thanks for your message, I have been able to run the parser that you are trying locally.
Does your harvest work or is it just the preview that is broken? Also what configuration details do you have for the API, Manager, and Worker endpoints in your respective application.yml files?
Just a note that the preview uses the details for your staging worker.
Thanks, Richard
Thanks @Reinkaos!
The harvest fails as well, but without errors in the 'harvest progress', but 'zeroes' in the 'number of records' line.
I've included my 'application.yml' files:
/manager/config/application.yml
production:
WORKER_HOST: http://127.0.0.1:3002
API_HOST: http://127.0.0.1:3000
HARVESTER_API_KEY: LKisxbcTwynxxd3ssyvn
API_MONGOID_HOSTS: localhost:27017
AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
WORKER_KEY: 7fdc382fc0bee62752f8cab38a0a04dbfb6c4535bdaf585a55e7eace560d64aeebf5f495edb628e35e739bedb0432580ddef23356e4bbf091acbb278b8a16111
#36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
development:
WORKER_HOST: http://127.0.0.1:3002
API_HOST: http://127.0.0.1:3000
HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
API_MONGOID_HOSTS: localhost:27017
AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
WORKER_KEY: W36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
test:
WORKER_HOST: http://127.0.0.1:3002
API_HOST: http://127.0.0.1:3000
HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
API_MONGOID_HOSTS: localhost:27017
AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
staging:
WORKER_HOST: http://127.0.0.1:4002
API_HOST: http://127.0.0.1:4000
HARVESTER_API_KEY: sd-vbuMeyfywzXaVaFqu
API_MONGOID_HOSTS: localhost:27017
WORKER_KEY: ade6ffdb44133c1efe3fc1e73560fd1649b561eef4e620e25b26757f095f36f244a2f042640264fcd89f1853236842b8a3559e0870c1c1600437fc2d44766b62
#WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
/worker/config/application.yml
development:
API_HOST: "http://127.0.0.1:3000"
HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
API_MONGOID_HOSTS: "localhost:27017"
MANAGER_HOST: "http://127.0.0.1:3001"
HARVESTER_CACHING_ENABLED: true
AIRBRAKE_API_KEY: "anc123"
LINK_CHECKING_ENABLED: "true"
LINKCHECKER_RECIPIENTS: "test@test.com"
WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
test:
API_HOST: "http://127.0.0.1:3000"
HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
API_MONGOID_HOSTS: "localhost:27017"
MANAGER_HOST: "http://127.0.0.1:3001"
HARVESTER_CACHING_ENABLED: false
AIRBRAKE_API_KEY: "anc123"
LINK_CHECKING_ENABLED: "true"
LINKCHECKER_RECIPIENTS: "test@test.com"
WORKER_KEY: YOUR_WORKER_KEY
staging:
API_HOST: "http://127.0.0.1:4000"
HARVESTER_API_KEY: sd-vbuMeyfywzXaVaFqu
MANAGER_HOST: "http://manager.bcdltestbed.ca"
HARVESTER_CACHING_ENABLED: true
AIRBRAKE_API_KEY: "anc123"
LINKCHECKER_EMAIL: "linkchecker@harvester.org"
LINK_CHECKING_ENABLED: "true"
HOST: 'worker.harvester.org'
SECRET_TOKEN: 'some long hash value'
DEVISE_MAILER: "info@harvester.org"
LINKCHECKER_RECIPIENTS: "bill@harvester.org"
API_MONGOID_HOSTS: "localhost:27017"
WORKER_KEY: ade6ffdb44133c1efe3fc1e73560fd1649b561eef4e620e25b26757f095f36f244a2f042640264fcd89f1853236842b8a3559e0870c1c1600437fc2d44766b62
#WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
production:
API_HOST: "http://api.bcdltestbed.ca"
MANAGER_HOST: "http://manager.bcdltestbed.ca"
HARVESTER_CACHING_ENABLED: true
AIRBRAKE_API_KEY: "anc123"
LINKCHECKER_EMAIL: "linkchecker@harvester.org"
LINK_CHECKING_ENABLED: "true"
SMTP_ADDRESS: "your SMTP address"
HOST: "worker.harvester.org"
SECRET_TOKEN: 'some long hash value'
DEVISE_MAILER: "info@harvester.org"
LINKCHECKER_RECIPIENTS: "bill@harvester.org"
API_MONGOID_HOSTS: "localhost:27017"
HARVESTER_API_KEY: LKisxbcTwynxxd3ssyvn
WORKER_KEY: 7fdc382fc0bee62752f8cab38a0a04dbfb6c4535bdaf585a55e7eace560d64aeebf5f495edb628e35e739bedb0432580ddef23356e4bbf091acbb278b8a16111
#36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77
/api/config/application.yml
defaults: &defaults
SOLR_PING: 'http://localhost:8982/solr/admin/ping'
HTTP_HOST: 'http://localhost:3000'
WORKER_API_URL: 'http://localhost:3002'
WWW_DOMAIN: 'www.dev'
DEVISE_MAILER: 'mailer@example.com'
RESQUE_USER: 'admin'
RESQUE_PASS: 'password'
HARVESTER_IPS: '127.0.0.1'
REQUEST_LIMIT_MAILER: 'develop@digitalnz.org'
BLACKLIST_MAILER_TO: 'blacklist@mailer.com'
BLACKLIST_MAILER_FROM: 'mailer@blacklist.org'
HTTP_HOST: 'http://test.host'
MONGOID_HOSTS: 'localhost:27017'
SECRET_TOKEN: '4a843fb4303933c16c1aacb057f976546315b620'
development:
<<: *defaults
test:
<<: *defaults
SOLR_PING: 'http://test.host'
HTTP_HOST: 'http://test.host'
WWW_DOMAIN: 'test.host'
staging:
<<: *defaults
production:
<<: *defaults
If the preview using the staging worker, I'm assuming I need to run a staging worker (a staging api too?) sorry . . .coming from your docker/dev builds this is new to me :)
Thanks again !
Hi @telezoic,
Sorry if that was confusing, the preview will use the worker of the RAILS_ENV that you are running the manager in. For instance if you run the manager in staging
mode it will use the staging worker for previews, otherwise if you run it in production
mode it will use the production worker for previews.
One other thing is that when a harvest runs the Worker will pull the parser script from the Manager over active resource. If you go into the rails console on the worker app, bundle exec rails c
. Can you find the parser your are attempting to run? You should be able to do something like this Parser.find(<id-of-your-parser')
. Where you can get the id of the parser from the manager on it's parser page.
Thanks, Richard
Thanks for your help with this @Reinkaos!
I found the parser with bundle exec rails c production
I took a closer look at my application.yml
files, made some adjustments, and restarted everything.
I can now successfully harvest and see the previews with a variety of parsers.
The one (or two) things I can't do however: are suppress a collection or suppress/delete a record in the relevant sections in the manager.
When I try and suppress a collection the collection status page remains blank below the 'suppressed collections" section. And the record update status page tells me my action is successful, but the record remains (even after a manual re-index).
I'm trying to trace how these events take place in the hopes it will help me troubleshoot this issue. Do you have any suggestions :) ?
Thanks,
Dan
Hi @telezoic,
I'm glad that your harvest is working now.
For the suppress collection the manager makes a request to the API on the route /harvester/sources/:source_id to update the source on the API side. Sources then get excluded by Solr which you can see on the API if you do a request like so:
http://<your-api>/records.json?api_key=<your-api-key>&debug=true
At the bottom of the page you will see the Solr query and it should have something like this in it:
solr_request_params: {
fq: [
"type:Record",
"record_type_i:0",
"-source_id_s:<your-source-id>",
null
],
This does not require a commit from Solr.
For the specific record, the manager makes a request to /harvester/records/:record_id on the API and will update the status. Records then have a call back on them after they have been saved which will remove the record from the index if it the record is not active. Solr will then need a commit for the changes to take affect.
I would have a look at the API logs and see if there are any errors first. The API needs to have the same sources as the manager so if they do not exist on the API then that would be another place to look. If you resave the source on the Manager it will be posted back to the API.
Hope that helps, Richard
Hi Richard,
This is super helpful! Thanks again.
There were no sources on the api side.
from mongo:
switched to db supplejack_api_production
> show collections
partners
preview_records
preview_sequences
record_interactions
records
sequences
supplejack_api_request_metrics
users
And my manager gave me a updating sources page error
"We're sorry something went wrong "
And the manager log, showed me these errors:
app/models/partner.rb:26:in `block in update_apis'
app/models/partner.rb:23:in `each'
app/models/partner.rb:23:in `update_apis'
app/models/source.rb:28:in `update_apis'
app/controllers/sources_controller.rb:44:in `update'
I swapped the API_HOST
variable in app/models/partner.rb
from:
RestClient.post("#{env['API_HOST']}/harvester/partners", . . .
to:
RestClient.post("http://127.0.0.1:8190/harvester/partners", . . .
and I can now suppress collections without difficulty!
My individual record suppression/deletion is still problematic however.
My solr install is set to autocommit.
<autoCommit>
<!--<maxTime>60000</maxTime> -->
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
I can update the record status and have the page confirm the record has been updated, and from the manager log it looks fine
method=GET path=/production/collection_records format=html controller=CollectionRecordsController action=index status=200 duration=60.70 view=5.84 params={"utf8"=>"✓", "commit"=>"Search Record", "environment"=>"production"} time=2019-09-05 21:04:47 +0000
method=PUT path=/production/collection_records/5 format=html controller=CollectionRecordsController action=update status=302 duration=106.62 view=0.00 location=http://manager.bcdltestbed.ca/production/collection_records?id=5 params={"utf8"=>"✓", "_method"=>"put", "authenticity_token"=>"r3/Pabbek4bXUkHvwGDuwN+lzSRLPuXusg+4G0p31AbI7wDPeGNpzva6vKyLEGFd07Nf8/gM3/Je/1hGAvOX+w==", "status"=>"deleted", "environment"=>"production"} time=2019-09-05 21:04:52 +0000
method=GET path=/production/collection_records format=html controller=CollectionRecordsController action=index status=200 duration=70.61 view=6.99 params={"environment"=>"production"} time=2019-09-05 21:04:52 +0000
and in the api log:
D, [2019-09-05T21:04:52.583825 #25152] DEBUG -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] MONGODB | 127.0.0.1:27017 | supplejack_api_production.update | STARTED | {"update"=>"records", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47232225922860 type=uuid data=0x1e20b7f7f4f3462c...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d5f813b2a041820f874fc80')}, "u"=>{"$set"=>{"status"=>"deleted", "updated_at"=>2019-...
D, [2019-09-05T21:04:52.594271 #25152] DEBUG -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] MONGODB | 127.0.0.1:27017 | supplejack_api_production.update | SUCCEEDED | 0.010s
I, [2019-09-05T21:04:52.612972 #25152] INFO -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] Completed 204 No Content in 76ms
But here's the part that's really throwing me. In an open search, my deleted record is visible
http://api.bcdltestbed.ca/records.json?api_key=LKisxbcTwynxxd3ssyvn
... {
"internal_identifier": "https://www.inaturalist.org/observations/10181692",
"created_at": "19/23/08",
"updated_at": "2019-09-05T21:04:52.582Z",
"record_id": 5,
"title": "Horse Mussel",
"description": null,
"display_collection": null,
"display_content_partner": [
"South Pacific Surprise!"
],
"category": [
"image"
...
but if I target the individual record: http://api.bcdltestbed.ca/records/5.json?api_key=LKisxbcTwynxxd3ssyvn
I get
{
"errors": "Record with ID 5 was not found"
}
Naturally I'd like to see the record gone from the open search :)
I'm thinking I'm missing a step, but I'm not sure what it is . . . something with solr?
I'd be grateful for any wisdom you could share.
Thanks,
D.
Hey @telezoic,
My suspicion is that you aren't running the cron which reindexes the records, can you confirm that you are running this cronjob? We run the cron every minute.
bundle exec rails runner -e production 'SupplejackApi::IndexRemainingRecordsInQueue.perform_async'
Also instead of removing the API HOST environment variable, just change the value in the application.yaml :).
Thanks, Richard
Hi folks,
I'm working through the production build as outlined at: https://digitalnz.github.io/supplejack/start/production-install.html
I think I've found a way (almost) to connect a production solr instance to the API. With solr 5 I have created a ‘production’ core, generated the sunspot solr files, copied those files to the production core, changed the sunspot.yml to point to production core, and created a symlink from the sunspot production folder to the solr production core.
I can start my solr instance and see my new production core [at localhost:8983/solr] by running
rake sunspot:solr:start RAILS_ENV=production
from the api folder.From the api folder if I run
bundle exec rails s -b 0.0.0.0 -p 3000
, the api gives the following error:I'm thinking this is because I haven't specified the production environment (8983)
I added a line to the puma.rb file
environment ENV.fetch("RAILS_ENV") { "production" }
and then tried starting the api with
bundle exec rails s -e production -b 0.0.0.0 -p 3000
.this throws the following:
I see some discussion online of a bug with ruby 2.5.+ (I'm at ruby 2.5.3p105) and devise 4.3.0 around the session controller. Updating to devise 4.4.0 removes the error but throws a new one from the api gem:
I hoping you might have a strategy/pointer to help me get around this :)
Thanks,
D.