Need a way to monitor custom mappings are in place

[ahafele]

We are finding that the default mapping gets put back in place unexpectedly - maybe from reinstalling modules or could even just doing an accidental PUT or /mapping-rules/{recordType}/restore... So we need a way to monitor ahead of migrations and in future to make sure it is in place for importing records.

Current custom mappings exist for

mapping-rules/marc-holdingsand mapping-rules/marc-bib

Options

• Daily script to check for particular fields?
• Use the mapping snapshots in some way?

[jermnelson]

Suggest create a new Airflow DAG that runs daily and checks the Okapi endpoints mentioned above for marc-holdings and marc-bib. These current mappings are stored in a Google Drive but they could be managed through the https://github.com/sul-dlss/folio_migration/ repository as resources in the mapping_files directory

[ndushay]

How about a warning (HoneyBadger, email, other) logged or sent when "restore" or PUT is done? This seems like something that may decrease a great deal when the churn dies down in (August).

[ndushay]

Or a way to check for mappings just before running a mapping ... as part of running a mapping?

[justinlittman]

What uses mapping rules? That is, if the custom mappings are replaced with defaults, what breaks?

[jermnelson]

The mod-data-import uses the mapping rules for importing MARC records to Instances. I'm not sure how we could hook-up Honeybadger or send an email to alert on an Okapi API PUT or restore call.

[justinlittman]

Is this only needed during migration or permanently? What is the context in which this is invoked (always Airflow?)?

[jermnelson]

This is needed permanently. We're planning to use the Data Import App for importing vendor loads and the Data Import App will be invoked by both Airflow and through the FOLIO UI.

[jmartin-sul]

• we can probably do this monitoring by running a scheduled DAG. writing that DAG is part of this ticket.
  • alternatively? additionally?: make this a check that runs at the start of an import DAG, and stops the rest of the DAG from running if error. but probably still want to monitor proactively, so we know if there's an issue before someone tries to run an import?
• if there's an issue discovered by the monitoring DAG, we can send an email to... a list... which may or may not exist already. or, could use honeybadger. HB might be best for consistency w/ other apps in dept.
• if the monitoring DAG fails unexpectedly, can configure an error email address (via notify_on_failure email option for the DAG, which is built into Airflow). may be a way to hook into HB for that, but that might be more fragile and more work to implement.

is this something other institutions need? something we could contribute back to folio? other institutions haven't complained about this, to our knowledge.

also, why is this happening? doesn't seem random that mappings get dropped, but cause isn't always unclear.

alternative idea from JLitt: something something nagios? maybe the check code could be written in such a way that it's invokable from either a script, by nagios, or from an airflow task at the start of a DAG. then, no new monitoring DAG. related idea from aaron: okcomputer check in dataloading app.

THE ACTION FOR THIS TICKET: is to figure out the options, pick one that looks esp promising, and perhaps ticket implementation. The first step might be a meeting to suss out options.

[jmartin-sul]

observation from laura: this applies to all uses of data import, e.g. also from ETD app.

[justinlittman]

The approach I'm going to take is to add a health check application to airflow (similar to CircRulesTester). This can then be monitored by Nagios and may be extended for other healthchecks.

[justinlittman]

Done.