Create a tool for sending avalanche reports emails

[asaunier]

Sympa, the mailing list system we used to send avalanche reports by email can no longer be used.

We need an alternate tool that we coud plug between:

• the avalanche reports subscription database
• the script the retrieve avalanche reports every day
• our third-party mailing service

[asaunier]

How it was done in v5:

• a job was scheduled several times a day to retrieve avalanche forecast reports from MF and SLF websites
• the job was managed by a python script https://github.com/c2corg/nivo-bulletins/blob/master/meteofrance%2Bslf.py (with phantomjs subscripts to get images etc.)
• the python script generated the emails and sent them to Sympa, the mailing lists server
• Sympa had access to the DB and to the tables of subscriptions, retrieving the lists of recipients for each avalanche forecast report, and forwarded the messages to the recipients using an SMTP server.
• Subscriptions were managed using the website interface, feeding special tables in the database (this is still the case with v6 and already done).

What has changed:

• Our version of Sympa was really old (>10 years) and incompatible with our new setup (eg. no longer compatible with our postgresql database server) and not upgradable because newer versions are not compatible with our usage => Sympa is off.
• The SMTP server is off. We now use the email service SparkPost https://www.sparkpost.com/docs/ - https://developers.sparkpost.com/api/

What we have to do:

• Update the report importing scripts to make it use the SparkPost service (retrieving recipients from the v6 API database) instead of sending emails to Sympa.
• Adaptations to the new environment (python 3) as well as phantomjs changes (+ updates of the MF and SLF websites?) might be required/useful.
• The script perhaps has to be integrated to the v6_api, eg. to easily access the database as well as the email service.

Example of third-party script (with DB access) in v6_api: https://github.com/c2corg/v6_api/blob/merge-users/c2corg_api/scripts/users/merge.py

Example of email sending in v6_api (registration and password change emails):

• https://github.com/c2corg/v6_api/blob/master/c2corg_api/views/user_account.py
• https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py

[asaunier]

Sending emails in v6

The v6_api app has a built-in email service, defined at https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py

The email service settings (including third-party email service host and API key etc.) are located in https://github.com/c2corg/v6_api/blob/master/common.ini.in#L79-L92 The actual settings values are set by default in https://github.com/c2corg/v6_api/blob/master/config/default#L66-L79 and overridden in demo and prod in the docker-compose.yml files (eg. the SparkPost data).

Sending emails is then quite easy, eg.:

from c2corg_api.emails.email_service import get_email_service
# ...
email_service = get_email_service(self.request)
# ...
email_service.send_change_email_confirmation(user, link)

See https://github.com/c2corg/v6_api/blob/master/c2corg_api/views/user_account.py#L111 for the actual code. It would probably be required to create a new method in the EmailService class like https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py#L86

I need to find out if one should call the email service method for every email address registered to the avalanche forecast reports or if it is possible to send group/batch emails. (edit: see just below for an answer)

v6_api's EmailService is based upon pyramid_mailer: https://docs.pylonsproject.org/projects/pyramid_mailer/en/latest/ => it is possible to pass a list of emails as the recipients property of the Message object. See https://docs.pylonsproject.org/projects/pyramid_mailer/en/latest/#pyramid_mailer.message.Message

Note that:

• using the current EmailService we will most likely have problems with free.fr addresses as for the registration emails or the password changing emails. See issue https://github.com/c2corg/v6_api/issues/700
• a python client exists for the email service we use, SparkPost. Perhaps using this client would help both solving the free.fr issue and using advanced email sending features (batch?): https://github.com/SparkPost/python-sparkpost A downside is that it would create a dependency on this service (would be harder to switch to some other service)

[asaunier]

Concerning SparkPost, the python client https://github.com/SparkPost/python-sparkpost offers interesting features. For instance:

• saved lists of recipients, we could use to store registered emails without using a table in our database: https://python-sparkpost.readthedocs.io/en/latest/api/recipient_lists.html
• message templates: https://python-sparkpost.readthedocs.io/en/latest/api/templates.html
• email sending function, accepting either a saved list of recipients (see above) or a list of emails (including as BCC): https://python-sparkpost.readthedocs.io/en/latest/api/transmissions.html#sparkpost.transmissions.Transmissions.send

For the last feature, please note we can already send emails to a list of email addresses using our existing pyramid_mailer tool: https://docs.pylonsproject.org/projects/pyramid_mailer/en/latest/#pyramid_mailer.message.Message

[asaunier]

According to the info above it looks possible to create a python script in the API (a bit like https://github.com/c2corg/v6_api/blob/merge-users/c2corg_api/scripts/users/merge.py) that would:

• get reports from SLF and MF website as done in https://github.com/c2corg/nivo-bulletins/blob/master/meteofrance%2Bslf.py
• request our database to get the list of recipients for a given avalanche report
• pass the message and recipients to SparkPost using either our existing EmailService or the SparkPost python client

@lbesson What do you think? You could explain how the report importing script is triggered (cron?) and how it figures out when a new report has been published? Does it store reports locally every time it connects to the source website and then compare them to detect new reports?

[asaunier]

Number of subscribers per list:

c2corg=# select count(user_subscriber), list_subscriber from sympa.subscriber_table group by list_subscriber;
 count |   list_subscriber   
-------+---------------------
   131 | meteofrance-66
   161 | meteofrance-64
   612 | meteofrance-05
  1027 | meteofrance-74
   164 | valanghe
    83 | lawinen
    42 | meteofrance-2b
   896 | meteofrance-38
   166 | meteofrance-06
   134 | meteofrance-31
    34 | meteofrance-2a
   258 | meteofrance-65
    28 | avalanche.en
    75 | aran
   102 | meteofrance-andorre
   176 | meteofrance-04
    61 | catalunya
   166 | meteofrance-09
   853 | meteofrance-73
   708 | avalanche
(20 rows)

[AntoineMarnat]

With Yeti, is this issue still relevant ? Or should it be closed ?