Notification updates?

[reginafcompton]

According to our MailChimp dashboard, notification emails have not gone out since January 8 at 6:05pm.

However:

• people are able to create user accounts and receive a confirmation email.
• the send_notifications logs indicates that emails are being sent.

[reginafcompton]

How to debug this:

Visit the django-admin view with a is_staff and is_superuser account. Look at the Redis queue: https://laws.council.nyc.gov/django-rq/queues. (It will show all the failed jobs and provide error codes.)

In this case, the error code for all failures (given a random sample) is: Work-horse process was terminated unexpectedly (waitpid returned 256)

This might suggest a memory error: https://stackoverflow.com/questions/47154856/work-horse-process-was-terminated-unexpectedly-rq-and-scrapy

I cleaned out the old failures log (i.e., those from 2017), and restarted redis. After which, I could send a single notifications email, triggered by the django-admin interface....but does the send_notifications command perform as expected?

Currently, investigating this.....

[reginafcompton]

Solution

We had three Councilmatics (the old nyc, the old nyc-staging, and new nyc) pointing to the same database:

RQ_QUEUES = {
    'default': {
         ...
        'DB': 1,
        ...
    }
}

The other Redis workers were still running (though cron tasks have been turned off). Turning off the unneeded redis workers solved the memory overload.

(We also ran sysctl vm.overcommit_memory=1, as instructed by the Redis log /var/log/redis/redis-server.log....that did not seem to do anything, but it's worth documenting here.)