Mistaken ingest

[hackartisan]

Summary

A bulk ingest was initiated for the studio_new directory. The resources created need to be deleted and the jobs need to be killed.

When I looked at the default queue this morning the jobs went up to #26295. the last couple are UpdateDao jobs. Before that I see jobs that look related to the large ingest, as well as a lot of derivative and characterization jobs, but we probably want to look more closely in the console.

the high queue has 4 ocr request jobs enqueued. the low queue has 375 jobs enqueued, probably all preservation jobs. super_low has 42,758.

Workers were turned off manually via the UI around 3:52 am on Tuesday morning, 5/28. There may be ingests or other jobs enqueued after that time, which we should try not to delete.

• [x] Figure out which jobs to remove and kill them
• [ ] Figure out which resources to delete and delete them.

Impact

Workers are currently off and until we can turn them back on figgy is essentially down for ingest.

Priority recommendation

• [x] asap
• [ ] within the next 3 weeks
• [ ] PO will prioritize

Sudden Priority Justification

Nothing can go into figgy right now.

[hackartisan]

Looking through some logs.

• bulk ingest post requests in the past week
  • This one enqueued studio_new/DPUL, with timestamp (Eastern time) May 23, 2024 at 5:17:23.000 pm. (That's Thursday).
• Here's all the IngestFolderJobs enqueued since that time

[hackartisan]

I've deleted the following jobs:

[{"enqueued_at"=>"2024-05-23T21:17:20Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/jennifer", "jid"=>"d16dd8bca8bfe846a877346b"},
 {"enqueued_at"=>"2024-05-23T21:17:21Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/will", "jid"=>"3a5aebf70e005ac09c75131b"},
 {"enqueued_at"=>"2024-05-23T21:17:20Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/jordan", "jid"=>"8859d74e543914a0e8eb420a"},
 {"enqueued_at"=>"2024-05-23T21:17:21Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/eal_ncl", "jid"=>"77613df1c7e1ead4cfa5f6ee"},
 {"enqueued_at"=>"2024-05-23T21:17:23Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/mary", "jid"=>"2b46722bbcd1549d318acb60"},
 {"enqueued_at"=>"2024-05-23T21:17:20Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/squirrel", "jid"=>"0e9a35af7c16fb295fb9013d"},
 {"enqueued_at"=>"2024-05-23T21:17:23Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/beth", "jid"=>"522bc0114e13311a98770ddf"},
 {"enqueued_at"=>"2024-05-23T21:17:23Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/zuzanna", "jid"=>"98474e5be5c68f2739f9c0ee"},
 {"enqueued_at"=>"2024-05-23T21:17:20Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/richel", "jid"=>"f39db2262d507c599b48ad67"},
 {"enqueued_at"=>"2024-05-23T21:17:21Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/microfilm", "jid"=>"6506cd737235603dbce3cef2"},
 {"enqueued_at"=>"2024-05-23T21:17:21Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/to_delete", "jid"=>"254da7d55b298dd5dd46a977"},
 {"enqueued_at"=>"2024-05-23T21:17:23Z", "directory"=>"/mnt/hydra_sources/studio_new/DPUL/sherry", "jid"=>"9205a8abe506f74b7620c0f2"}]

[hackartisan]

I think the ingester deleted the resources that successfully made it through the ingest process. I think this is safe to close.

[hackartisan]

update: this morning right now there are 20,125 jobs in the default queue. super_low has > 88,000.