Known work around Bug: Email delay-previously emails generated by examining a name had a 5-minute delay.

[ozamani9gh]

Priority 3: Email delay-previously emails generated by examining a name had a 5-minute delay. This was helpful when examining as any mistakes (forgot to put condition on, made error in examining, etc.) wouldn’t result in multiple outputs that are confusing for the client. Preference for a short delay that only generates the last email. (Frequency 100%) Time cost: approx. 20-30min/week

[ozamani9gh]

Update from Joshua: • The removal of the output delay creates confusion for clients if an examiner were to catch a mistake and go back to fix it (generates multiple different results emails to clients, who then call or email in confusion which creates a cascade effect). Even a few minutes delay for output generation would help reduce this.

[ozamani9gh]

@Mihai-QuickSilverDev reach out to Joshua and get the exact delay he wants, we suggest 10 or 15 minutes.

[Mihai-QuickSilverDev]

Message sent to Joshua on June 28, 2024, 9 AM:

Hi Joshua,

Regarding this topic below, could you please let us know what would be a desirable duration for this delay? We could do 5, 10, 15 minutes, whatever would best fit the Name Examiners needs.

Update from Joshua: • The removal of the output delay creates confusion for clients if an examiner were to catch a mistake and go back to fix it (generates multiple different results emails to clients, who then call or email in confusion which creates a cascade effect). Even a few minutes delay for output generation would help reduce this.

Thank you,

[ozamani9gh]

Joshua stated 5 minutes is sufficient

[EPortman]

Clients are emailed about their NRs in the following scenarios:

1. When NR goes to APPROVE, CONDITIONAL or REJECTED
2. When a payment update is made to REAPPLY or UPGRADE
3. When a NR is refunded
4. When a NR updates to CONSENT_RECEIVED

[EPortman]

Async Operations with Flask Framework: https://flask.palletsprojects.com/en/3.0.x/async-await/# Async Operations with Gunicorn Server: https://docs.gunicorn.org/en/latest/design.html

NameX API uses the Flask Framework running on a Gunicorn WSGI Server

Why its not possible to us async/await on each email for 5 minutes Flask on its own is single-threaded (blocking) and does not natively support asynchronous operations across requests - only within them (can await a database operation, but cant schedule it - it will be cancelled once the request ends since flask handles 1 request at a time).

• In production, a Gunicorn WSGI Server allows Flask to be concurrent (multiple requests at once)
• Gunicorn employs several instances of the app called workers that run independently of each other
  • These workers can be sync (default) or gevent (green threads)
  • sync is the default worker type and the one that is used in our API
    • In this scenario, each NR needs to wait for 5 minutes so it needs to schedule an async task that outlives itself. In the Flask Link provided, under Background Tasks, it states that this is not possible on a WSGI server (1 request-response cycle per worker).

To get the desired behavior (non-blocking waiting for 5 minutes for each email)

1. Change the Gunicorn worker to be of type gevent to allow for async operations

   • This would change the way the Server works - it would support asynchronous operations (have an event loop, gEVENT)
   • The section When to use Quart instead in the Flask docs seem to imply this is more of a work-around.

2. Spawn a separate thread for async operations

   • Spawn a separate thread with an event loop to handle all asynchronous tasks.
     • This approach adds a layer of complexity - API would become multithreaded with a main sync thread and another async thread where emails tasks are scheduled.

3. Use Celery to get the async effect

   • Celery is a separate process that acts as a task queue.
   • API would send asynchronous tasks to this separate process.
   • The recommended way, involves adding another dependency and separate running process to the api .

4. Use a Cron Job to get the async effect

   • Any time an email needs to be sent, it is written to disk in the database.
   • A cron job then reads from this database every 5 minutes (or even at the end of every day) and determines what emails need to be sent.

[EPortman]

A good resource for this topic: https://testdriven.io/blog/flask-async/

[EPortman]

It is not really an option to change the functionality of the queue. The queue module is located here: https://github.com/bcgov/namex/blob/main/services/pubsub/gcp_queue/pubsub.py

Reasons why changing this queue is not an option

• A lot of other services use this queue

• The queue also runs on a Flask Server and has the same problem of not being able to perform async operations.

  • It cannot wait for 5 minutes in between requests as this would block the queue for 5 minutes or the task would be cancelled once the request ends. This is the code that the API calls to publish emails:

    def publish(self, topic: str, payload: bytes):
    """Send payload to the queue."""
    if not (publisher := self.publisher):
        raise Exception("missing setup arguments")  # pylint: disable=W0719
    
    try:
        future = publisher.publish(topic, payload)  # Send email to gcp emailer
    
        return future.result()                      # Wait for the completion and then move on
    except (CancelledError, TimeoutError) as error:
        raise Exception("Unable to post to queue", error) from error  # pylint: disable=W0719

• Trying to change it here (the queue service) would be the same problem with changing it in the api. Would still have to have a way of getting an event loop to be able to use async/await across requests or using a separate process (celery) to get the async effect and not have the 5 minute wait for each email (blocks the server).