Harvester Email

[jbrown-xentity]

User Story

In order to notify data providers of harvest job completion and/or errors, data.gov data providers want an email sent out on harvest job completion.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [ ] GIVEN the email is extractable from the CKAN organization \ WHEN a harvest job occurs for a given harvest source \ THEN an email is sent with a summary to the emails specified above \ AND there is some way to digest the problems (whether in the email, or through a link)

Background

Popped into my brain from this comment, what do we do with errors. This is what we do.

Security Considerations (required)

No security considerations. Some data providers may balk at their harvests being publicly available

Sketch

No idea at this point. Need to investigate airflow options and best practices, possibly utilizing the SES cloud.gov service (to keep email from point consistent). If there are no out of the box items, change this to a spike/discovery ticket on how to approach this problem.

[nickumia]

Maybe useful references when the time comes:

• https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html
• https://docs.astronomer.io/learn/error-notifications-in-airflow

[rshewitt]

flask mail extension

[btylerburton]

Agree for MVP to send a link to the Harvest UI Jobs page

[Jin-Sun-tts]

Initially, consider using the flask-mail plugin, which allows sending emails directly from a Flask application. However, our requirement is to incorporate the email step into the harvest process as the final action within the report step.

If we use flask-mail, we would need access to the application instance to send emails. This could lead to redundancy in app instances or require creating an API route that the harvest runner would call, potentially increasing network traffic.

Instead, working on Python's built-in smtplib library to send emails directly from the harvest process. This approach avoids the need for a Flask app instance and eliminates the overhead of creating an API route, reducing potential network traffic and simplifying the integration.

[Jin-Sun-tts]

The email function is now woking in development, sample email like below:

Subject: Harvest Job Completed