Generate a Data Export Report and send Email - DAG failure

[ahafele]

When an export DAG (select OR send) has failed (and will not be retried) email should go out to folio-load-reports@lists.stanford.edu and sul-unicorn-devs@lists.stanford.edu with:

Subject/body:

"Vendor Name: A data export has failed ..."

• Vendor name
• on Date
• Link to DAG failure

• Link to UI for downloading mrc or reports

  For example, orafin dags generate emails.

No report needed for FULL DUMP

Acceptance Criteria

Generated by Zenhub AI.

• [ ] Scenario: Generate Data Export Report and send Email - DAG failure
  • Given a DAG has failed (and will not be retried)
  • When an email is sent to folio-load-reports@lists.stanford.edu and sul-unicorn-devs@lists.stanford.edu
  • Then the email should have the following content:
    1. Subject: "A data export has failed ..."
    2. Vendor name
    3. Date of failure
    4. Link to DAG failure
    5. Link to UI for downloading mrc or reports

[jgreben]

We should maybe have a story time about this...

As I see it, currently if the selection dag fails, it will not be retried for the same day, so we should always send out an email for a failed selection dag.

For the transmission dag failures there is already an email that is sent to sul-unicorn-devs each time there are failures in the file list. The failures do not get archived and will keep getting re-tried every day until someone clears them out, so there is no current state kept about the re-tries, and they will always be re-tried.

So maybe for this ticket we just need to send a notification about a failed selection dag, and maybe reformat the failed transmission email?

[ahafele]

Can you point me to an example of the transmission dag failure email?

[jgreben]

@ahafele https://github.com/sul-dlss/libsys-airflow/blob/e59f59ccbf4737e740cabdbc55b75c2e3a2c4c34/libsys_airflow/plugins/data_exports/email.py#L78

[jgreben]

It will basically look something like this example:

Failed to Transmit Files for send_pod_records scheduled__2024-06-26T07:00:00+00:00

scheduled__2024-06-26T07%3A00%3A00%2B00%3A00

These files failed to transmit data-export-files/some-vendor/marc-files/new/2024062801.mrc data-export-files/some-vendor/marc-files/updates/2024062802.mrc data-export-files/some-vendor/marc-files/updates/2024062803.mrc

[ahafele]

This looks good to me.

[jgreben]

Update: also send email when a selection dag fails.

Test that transmission fail email works.

[jgreben]

New ticket for emailing selection dag failures. Closing.