gaurijo
commented
10 months ago I'm currently working on getting the Hathitrust check automated after import is complete (in order to get this working locally, I had to tinker with the code to bypass the need to be in production or demo mode). The code in the PR will be slightly different and less messy, but I wanted to document how I tested it:
The import_async(task) method includes code to instantiate a new hathitrust object, and then calls on that object to run the .check(task) method (outside of testing purposes, this is going to be called on with .check_async(task) ):
app/models/record_source.rb
def import_async(task)
unless Rails.env.production? or Rails.env.demo?
raise 'This feature only works in production. '\
'Elsewhere, use a rake task instead.'
end
hathi_trust = Hathitrust.new
hathi_trust.check(task)
## rest of code
endI required the path to the Hathitrust model inside the Task model, and then included two new methods for testing the automated hathitrust check triggers --
- the
import_task_completed?which checks if theimport status is failed(which, during testing it always will be, due toaws token expiring) and if the service is for local storage. In prod/demo the condition will be changed so it checks if the import status succeeded. - If that condition passes, then the second method should start the hathitrust check (
trigger_hathitrust_check). This is processed as aafter_savecallback to ensure the object is saved before trigger logic. Again, for local testing purposes, the action called on inside this method is.check(task)but otherwise it should be.check_async(task):
app/models/task.rb
require_relative './hathitrust.rb'
class Task < ApplicationRecord
class Status
## original code
end
after_initialize :init
before_save :constrain_progress, :auto_complete
after_save :trigger_hathitrust_check, if: :import_task_completed?
def import_task_completed?
status == Status::FAILED and service == Service::LOCAL_STORAGE
## status is always FAILED when running import locally bc of aws tokens
end
def trigger_hathitrust_check
hathi_trust = Hathitrust.new
task = Task.create!(name: 'Preparing to check HathiTrust',
service: Service::HATHITRUST,
status: Status::SUBMITTED)
hathi_trust.check(task)
end
Inside the Hathtrust model, I commented out the class method Book.analyze_table for testing locally. Including the method was giving me a local error: PG::ActiveSQLTransaction: ERROR: VACCUM cannot run inside a transaction block
app/models/hathitrust.rb
class Hathitrust
def self.check_authorized?
## original code
end
def check(task = nil)
## original code
rescue => e
Rails.logger.error("Hathitrust.check(): #{e}")
task.update!(name: "HathiTrust check failed: #{e}",
status: Task::Status::FAILED)
raise e
else
task.name = "Checking HathiTrust: updated database with "\
"#{Book.where(exists_in_hathitrust: true).count} found items."
task.status = Task::Status::SUCCEEDED
task.save!
puts task.name
ensure
FileUtils.rm(pathname, force: true) if pathname.present?
# Book.analyze_table
end
end
## original code
## private methods
end
After making these tweaks, I was able to run bin/rails books:import, which scanned the records locally, and then triggered a HathiTrust check automatically:
adolski
Currently, each step in the import-and-check process is manual, invoked by a button click. We would like to modify this so that, once an import completes, the import step invokes all three of the checking steps, as ECS tasks in parallel. The end result is that there is only one button (the import button) left in the UI.