Handle FOLIO deletes in the indexer

[cbeer]

When a record is deleted from FOLIO (or marked as suppressed from discovery? or becomes otherwise invalid?), the record should be removed from Solr.

[thatbudakguy]

still grappling with FOLIO's distinction between instance record and source record, but it seems like it's important to understand in the context of deletes. to my knowledge:

• instance records are FOLIO's concept of a bib; they cannot be deleted — only suppressed. it's possible to create an instance record entirely within FOLIO without ever interacting with any MARC, in which case it won't have a source record (its source is FOLIO, as opposed to MARC or EPKB).
• source records are the actual MARC that was used to create an instance record. unclear if some properties of the instance record effectively delegate to its source record, or vice versa. there are a variety of ways to indicate that a source record is deleted.

for source records (MARC) at the /source-storage/stream/source-records endpoint:

• you can filter by suppressFromDiscovery. maybe this references the associated instance record?
• you can filter by deleted. this checks if any of the following are true:
  • the state of the record is DELETED
  • the state of the record is ACTUAL (?)
  • the MARC leader 05 is set to d, s, or x. marc21 provides d as "deleted"; unclear what s or x are for (or if they're valid MARC).

[thatbudakguy]

old PR? https://github.com/sul-dlss/searchworks_traject_indexer/pull/707

[thatbudakguy]

we need to confirm (with @ahafele) how records are "deleted" (marked suppressed, MARC edited, etc.) in FOLIO now so that we can cover all the cases and find some records to test with.

is it OK for us to mark things as suppressed, etc. for testing? are there good records for us to do this with?

[ahafele]

Right now records will be "deleted" by marking them as suppressed at the Instance level. Here is an example - https://folio-test.stanford.edu/inventory/view/0c7e5994-fce7-5a85-a715-384146ec3d3f You are welcome to mark additional records as suppressed for further testing or if you prefer I can grab a test set, mark them for SW testing and then mark them all as suppressed - any other criteria to consider?

We may consider adding a stat code for DELETE, but I assume SW only cares about suppression and not the reason for suppression. In the future there will be an explicit "mark for deletion" flag but not in Nolana.

Open related question on our end - how will SW handle instance records with ALL items suppressed - will it assume suppression at the instance level or will it need to be explicit?

[cbeer]

Open related question on our end - how will SW handle instance records with ALL items suppressed - will it assume suppression at the instance level or will it need to be explicit?

I think we'd strongly prefer explicit suppression, especially now that some instances (on-order, electronic, mhlds) are expected to lack items.

[jcoyne]

Test plan:

• find a record in https://searchworks-folio-dev.stanford.edu/
• mark it suppressed in folio-test
• after ~15 minutes~ 1hr15m (Postgres poll delay + Solr autocommit delay) ensure it can no longer be found in https://searchworks-folio-dev.stanford.edu/

[jcoyne]

Executed test plan (PENDING)

• Record https://searchworks-folio-dev.stanford.edu/view/14598667
• Marked suppressed in https://folio-test.stanford.edu/inventory/view/fb9341f5-7fc9-5a7a-b608-7d341768e7ce?qindex=hrid&query=a14598667&sort=title at 5:26 PDT 5/3/2023
• ~This record still resolved at 12:30 PDT: https://searchworks-folio-dev.stanford.edu/view/14598667 and the last_updated field was "2023-04-27T18:29:17Z" (http://sul-solr-prod-b.stanford.edu/solr/#/searchworks-folio-dev/query?q=id:14598667&q.op=OR&indent=true&fl=id,last_updated)~

I ran this query:

record = Traject::FolioPostgresReader.find_by_catkey('a14598667', 'postgres.url'=>'<uri>')
 record.instance['suppressFromDiscovery']
=> true

So it seems like the UI is setting the correct thing in the database.

[jcoyne]

This now appears to be working. Status update are showing up in searchworks ~20 min after the hour.