Generate a Data Export Report and send Email - DAG failure

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows

Apache License 2.0

5 stars 0 forks source link

Generate a Data Export Report and send Email - DAG failure #1079

Open ahafele opened 2 weeks ago

ahafele commented 2 weeks ago

When an export DAG (select OR send) has failed (and will not be retried) email should go out to folio-load-reports@lists.stanford.edu and sul-unicorn-devs@lists.stanford.edu with:

Subject/body:

"Vendor Name: A data export has failed ..."

Vendor name
on Date
Link to DAG failure
Link to UI for downloading mrc or reports

For example, orafin dags generate emails.

No report needed for FULL DUMP

Acceptance Criteria

Generated by Zenhub AI.

[ ] Scenario: Generate Data Export Report and send Email - DAG failure
- Given a DAG has failed (and will not be retried)
- When an email is sent to folio-load-reports@lists.stanford.edu and sul-unicorn-devs@lists.stanford.edu
- Then the email should have the following content:
  1. Subject: "A data export has failed ..."
  2. Vendor name
  3. Date of failure
  4. Link to DAG failure
  5. Link to UI for downloading mrc or reports

jgreben commented 1 week ago

We should maybe have a story time about this...

As I see it, currently if the selection dag fails, it will not be retried for the same day, so we should always send out an email for a failed selection dag.

For the transmission dag failures there is already an email that is sent to sul-unicorn-devs each time there are failures in the file list. The failures do not get archived and will keep getting re-tried every day until someone clears them out, so there is no current state kept about the re-tries, and they will always be re-tried.

So maybe for this ticket we just need to send a notification about a failed selection dag, and maybe reformat the failed transmission email?

ahafele commented 1 week ago

Can you point me to an example of the transmission dag failure email?

jgreben commented 2 days ago

@ahafele https://github.com/sul-dlss/libsys-airflow/blob/e59f59ccbf4737e740cabdbc55b75c2e3a2c4c34/libsys_airflow/plugins/data_exports/email.py#L78

jgreben commented 2 days ago

It will basically look something like this example:

Failed to Transmit Files for send_pod_records scheduled__2024-06-26T07:00:00+00:00

scheduled__2024-06-26T07%3A00%3A00%2B00%3A00

These files failed to transmit data-export-files/some-vendor/marc-files/new/2024062801.mrc data-export-files/some-vendor/marc-files/updates/2024062802.mrc data-export-files/some-vendor/marc-files/updates/2024062803.mrc