ualbertalib / jupiter

Jupiter is a University of Alberta Libraries-based initiative to create a sustainable and extensible digital asset management system. This is phase 2 (Digitization).
https://era.library.ualberta.ca/
MIT License
23 stars 10 forks source link

Removing a stale Solr result in production #3492

Closed pgwillia closed 1 month ago

pgwillia commented 1 month ago

View details in Rollbar: https://app.rollbar.com/a/ualbertalib/fix/item/jupiter/1918

Removing a stale Solr result, be28d90c-6a87-4547-bd06-399e5ca0d708: {"id"=>"be28d90c-6a87-4547-bd06-399e5ca0d708", "visibility_ssim"=>["http://terms.library.ualberta.ca/public"], "record_created_at_dtsi"=>"2024-06-11T19:41:22.338Z", "date_ingested_dtsi"=>"2024-06-11T19:41:22.338Z", "title_tesim"=>["Decoding the Heart"], "title_ssi"=>"Decoding the Heart", "sort_year_isim"=>[2024], "sort_year_isi"=>2024, "member_of_paths_dpsim"=>["e14a6354-b557-4a15-8e99-70ad237be121/c9f59edb-6d07-41bf-a324-9b6caca431c5"], "subject_tesim"=>["Research image"], "creators_tesim"=>["DasGupta, Ishani"], "created_tesim"=>["2024-06-01"], "created_ssi"=>"2024-06-01", "languages_tesim"=>["http://id.loc.gov/vocabulary/iso639-2/eng"], "license_tesim"=>["http://creativecommons.org/licenses/by-nc/4.0/"], "item_type_ssim"=>["http://purl.org/ontology/bibo/Image"], "has_model_ssim"=>["ArItem"], "owner_ssim"=>["1"], "description_tesim"=>["The image portrays the synergistic relation between the two main aspects of my research - medical image analysis and coding strategies. I dynamically engage with both, symbolized by the reflections seen within the frames. The aim is to efficiently and accurately identify the left ventricle (LV) from 3D heart ultrasound (echocardiography) images. \nBy examining heart images, I am developing a machine learning (ML) assisted automated system that can efficiently and accurately locate the LV and analyze heart function. Working with cardiologists at the University Hospital, I strive to improve the efficiency of identifying heart diseases. I design, code, and train neural networks that continuously learn the principal features in these heart images. Once the trained ML model has generated its predicted outputs, I use medical imaging software to view, examine, and compare these ML outputs with the expert LV identification.\nIn my research, the fundamental interplay between these two domains is essential for advancing the understanding and capabilities of cardiovascular imaging.\n"], "_version_"=>1801595009808465920, "timestamp"=>"2024-06-11T19:41:29.419Z", "score"=>1.0}

https://github.com/ualbertalib/jupiter/blob/66ca00d348625678803e91c56c69daf6ac8d6e5c/app/models/jupiter_core/solr_services/deferred_faceted_solr_query.rb#L146-L154

pgwillia commented 1 month ago

@murny says, "Looks like at ~1:41 PM yesterday something happen where a bunch of things couldn’t be found in the database 🤔. So these Solr errors are when we try to look up an object in ActiveRecord and it’s not found (we assume Solr is stale, so we remove the record from Solr). But this also happen in Sidekiq, where we saw a big uptick of ActiveRecord Not Found errors in DOICreateJob’s. It hasn’t happen again since 1PM yesterday (2024-06-11) and no more new errors for this. Don’t understand how this error could happen in a blimp like this though, unless we swung to a different DB for a few seconds or something 🤷 (cause if DB went down or went offline, you would get a different error (timeout or connection error, not record not found) These objects do exist though and I can view them no problem in the search results and show page in production as of right now"

Seems to be a one off and nothing is missing. We'll watch for this to re-occur.

pgwillia commented 1 month ago

@nmacgreg I wonder if this was related to IST resolving the PostgreSQL database backup/firewall the other day.