search

ebmdatalab / openprescribing

A Django app providing a REST API and dashboards for the HSCIC's GP prescribing data
https://openprescribing.net
MIT License
97 stars 26 forks source link

Alerts still not getting sent out without hassle #2517

Open sebbacon opened 4 years ago

sebbacon commented 4 years ago

Running the command with

./manage.py send_monthly_alerts --max_errors=50

        /webapps/openprescribing/openprescribing/frontend/views/bookmark_utils.py:205: RuntimeWarning: Mean of empty slice
  self.target_means.append(np.nanmean(window))
/webapps/openprescribing/.venv35/lib/python3.5/site-packages/numpy/lib/nanfunctions.py:1667: RuntimeWarning: Degrees of freedom <= 0 for slice.
  keepdims=keepdims)
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
Error posting to https://api.mailgun.net/v3/openprescribing.net/messages:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)
Sending a message to xxx@nhs.net from OpenPrescribing <feedback@openprescribing.net>
Traceback (most recent call last):
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
    cnx.do_handshake()
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1646, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred: 

Traceback (most recent call last):
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connection.py", line 360, in connect
    ssl_context=context,
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    raise timeout("select timed out")
During handling of the above exception, another exception occurred: 

Traceback (most recent call last):
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) 
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value  
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred: 

Traceback (most recent call last):
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base_requests.py", line 74, in post_to_esp
    response = self.session.request(**params)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/raven/breadcrumbs.py", line 341, in send
    resp = real_send(self, request, *args, **kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred: 

Traceback (most recent call last):
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base_requests.py", line 74, in post_to_esp
    response = self.session.request(**params)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/raven/breadcrumbs.py", line 341, in send
    resp = real_send(self, request, *args, **kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred: 

Traceback (most recent call last):
  File "/webapps/openprescribing/openprescribing/common/alert_utils.py", line 40, in try_email
    callback(*args)
  File "/webapps/openprescribing/openprescribing/frontend/management/commands/send_monthly_alerts.py", line 189, in send_org_bookmark_email
    msg.send()
  File "/webapps/openprescribing/openprescribing/frontend/models.py", line 1010, in send
    self.message.send()
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/django/core/mail/message.py", line 291, in send
    return self.get_connection(fail_silently).send_messages([self]) 
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base.py", line 95, in send_messages
    sent = self._send(message)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base_requests.py", line 60, in _send
    return super(AnymailRequestsBackend, self)._send(message)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base.py", line 125, in _send
    response = self.post_to_esp(payload, message)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/anymail/backends/base_requests.py", line 81, in post_to_esp
    raised_from=err, email_message=message, payload=payload)
anymail.backends.base_requests.AnymailRequestsAPIError: Error posting to https://api.mailgun.net/v3/openprescribing.net/messages:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)
Sending a message to xxx@nhs.net from OpenPrescribing <feedback@openprescribing.net>
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
console: [opttick] - GTM-5PX77GZ: exception: TypeError: undefined is not a function (evaluating 'r.getEntries()')
1   0x1b67967 /usr/local/bin/phantomjs() [0x1b67967]
2   0x1baac35 /usr/local/bin/phantomjs() [0x1baac35]
3   0x1b9369c /usr/local/bin/phantomjs() [0x1b9369c]
4   0x18970ad /usr/local/bin/phantomjs() [0x18970ad]
5   0x1896ace /usr/local/bin/phantomjs() [0x1896ace]
6   0x1896ddf /usr/local/bin/phantomjs() [0x1896ddf]
7   0x190663e /usr/local/bin/phantomjs() [0x190663e]
8   0x7fdbd5dfc635 [0x7fdbd5dfc635]
PhantomJS has crashed. Please read the bug reporting guide at
<http://phantomjs.org/bug-reporting.html> and file a bug report.
Segmentation fault
Command '/usr/local/bin/phantomjs /webapps/openprescribing/openprescribing/frontend/management/commands/grab_chart.js "https://openprescribing.net/analyse/#org=practice&orgIds=03L,02N,00T,99C,01A,00N,01C,00L,00Q,02W,01H,01K,03F,03A,03H,00V,08K,05D,00Y,01W,01Y,02A,06M,01J,01T,01V,12F,01X,09X,03D,04E,04C,15C,05Y,03Q,03T,05L,04G,03V,04H,04L,07W,08C,07Y,04K,07R,04Y,05N,05C,08P,05F,05Q,08L,07V,09D,05T,10C,10K,08H,06H,06L,06Q,10X,00R,02T,07L,00K,99A,06Y,07J,03J,10L,07K,07M,07Q,09G,10Q,10R,06P,10E,07G,02H,14L,11A,00C,04M,11J,07T,11M,03W,11N,11X,14Y,15M,07X,15N,99J,11E,99N,02E,00P,02M,99F,08W,08M,02F,08E,02P,03E,08V,08Q,03M,08T,08X,00X,04F,09E,09Y,04V,05R,05W,06F,06T,02Y,01R,06V,07H,05G,09N,10J,15E,08Y,05V,15A,08D,99D,08A,99K,05H,09A,09J,05X,09C,99E,00D,08R,15D,06N,99G,01E,01F,03R,06D,06K,02X,10D,09F,06A,02G,00M,06W,02R,03N,05A,01G,04N,03K,09W,02D,07P,02Q,13T,00J,07N,08N,15F,01D,04Q,05J,09L,09P,10V,04D,08G,08F,08J,10A,09H,12D,99M,99H&numIds=0109040N0BD&denom=nothing&selectedTab=summary" /mnt/database/tmp/tmphjo_ca8b.png "#results .tab-pane.active" 800x600 500' returned non-zero exit status 139
raceback (most recent call last):
  File "/webapps/openprescribing/openprescribing/common/alert_utils.py", line 40, in try_email
    callback(*args)
  File "/webapps/openprescribing/openprescribing/frontend/management/commands/send_monthly_alerts.py", line 199, in send_search_bookmark_email
    msg = bookmark_utils.make_search_email(search_bookmark, tag=now_month)
  File "/webapps/openprescribing/openprescribing/frontend/views/bookmark_utils.py", line 763, in make_search_email
    "#results .tab-pane.active",
  File "/webapps/openprescribing/openprescribing/frontend/views/bookmark_utils.py", line 503, in attach_image
    result = subprocess.check_output(cmd, shell=True)
  File "/usr/lib/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/usr/local/bin/phantomjs /webapps/openprescribing/openprescribing/frontend/management/commands/grab_chart.js "https://openprescribing.net/analyse/#org=practice&orgIds=03L,02N,00T,99C,01A,00N,01C,00L,00Q,02W,01H,01K,03F,03A,03H,00V,08K,05D,00Y,01W,01Y,02A,06M,01J,01T,01V,12F,01X,09X,03D,04E,04C,15C,05Y,03Q,03T,05L,04G,03V,04H,04L,07W,08C,07Y,04K,07R,04Y,05N,05C,08P,05F,05Q,08L,07V,09D,05T,10C,10K,08H,06H,06L,06Q,10X,00R,02T,07L,00K,99A,06Y,07J,03J,10L,07K,07M,07Q,09G,10Q,10R,06P,10E,07G,02H,14L,11A,00C,04M,11J,07T,11M,03W,11N,11X,14Y,15M,07X,15N,99J,11E,99N,02E,00P,02M,99F,08W,08M,02F,08E,02P,03E,08V,08Q,03M,08T,08X,00X,04F,09E,09Y,04V,05R,05W,06F,06T,02Y,01R,06V,07H,05G,09N,10J,15E,08Y,05V,15A,08D,99D,08A,99K,05H,09A,09J,05X,09C,99E,00D,08R,15D,06N,99G,01E,01F,03R,06D,06K,02X,10D,09F,06A,02G,00M,06W,02R,03N,05A,01G,04N,03K,09W,02D,07P,02Q,13T,00J,07N,08N,15F,01D,04Q,05J,09L,09P,10V,04D,08G,08F,08J,10A,09H,12D,99M,99H&numIds=0109040N0BD&denom=nothing&selectedTab=summary" /mnt/database/tmp/tmphjo_ca8b.png "#results .tab-pane.active" 800x600 500' returned non-zero exit status 139
Traceback (most recent call last):
  File "./manage.py", line 26, in <module>
    execute_from_command_line(sys.argv)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/webapps/openprescribing/.venv35/lib/python3.5/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/webapps/openprescribing/openprescribing/frontend/management/commands/send_monthly_alerts.py", line 255, in handle
    self.send_search_bookmark_email, search_bookmark, now_month
  File "/webapps/openprescribing/openprescribing/common/alert_utils.py", line 53, in __exit__
    raise exception
common.alert_utils.BatchedEmailErrors: Encountered 2 mail exceptions (showing last traceback only): `anymail.backends.base_requests.AnymailRequestsAPIError: Error posting to https://api.mailgun.net/v3/openprescribing.net/messages:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.mailgun.net', port=443): Read timed out. (read timeout=30)
Sending a message to xxx@nhs.net from OpenPrescribing <feedback@openprescribing.net>, subprocess.CalledProcessError: Command '/usr/local/bin/phantomjs /webapps/openprescribing/openprescribing/frontend/management/commands/grab_chart.js "https://openprescribing.net/analyse/#org=practice&orgIds=03L,02N,00T,99C,01A,00N,01C,00L,00Q,02W,01H,01K,03F,03A,03H,00V,08K,05D,00Y,01W,01Y,02A,06M,01J,01T,01V,12F,01X,09X,03D,04E,04C,15C,05Y,03Q,03T,05L,04G,03V,04H,04L,07W,08C,07Y,04K,07R,04Y,05N,05C,08P,05F,05Q,08L,07V,09D,05T,10C,10K,08H,06H,06L,06Q,10X,00R,02T,07L,00K,99A,06Y,07J,03J,10L,07K,07M,07Q,09G,10Q,10R,06P,10E,07G,02H,14L,11A,00C,04M,11J,07T,11M,03W,11N,11X,14Y,15M,07X,15N,99J,11E,99N,02E,00P,02M,99F,08W,08M,02F,08E,02P,03E,08V,08Q,03M,08T,08X,00X,04F,09E,09Y,04V,05R,05W,06F,06T,02Y,01R,06V,07H,05G,09N,10J,15E,08Y,05V,15A,08D,99D,08A,99K,05H,09A,09J,05X,09C,99E,00D,08R,15D,06N,99G,01E,01F,03R,06D,06K,02X,10D,09F,06A,02G,00M,06W,02R,03N,05A,01G,04N,03K,09W,02D,07P,02Q,13T,00J,07N,08N,15F,01D,04Q,05J,09L,09P,10V,04D,08G,08F,08J,10A,09H,12D,99M,99H&numIds=0109040N0BD&denom=nothing&selectedTab=summary" /mnt/database/tmp/tmphjo_ca8b.png "#results .tab-pane.active" 800x600 500' returned non-zero exit status 139`
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
sebbacon commented 4 years ago

I can see repeated (?) mailgun connection errrors, and a phantomjs core dump in there.

We really need to put some thought into making this run unsupervised!

My questions:

For now, I've just run the identical command again to see if it gets to the end.

sebbacon commented 4 years ago

Re-running exhausted the 50-error limit very quickly, implying that these are not transient errors. Now re-running with a 200-error limit.

sebbacon commented 4 years ago

I guess after the alert sending has completed, we can review items that didn't send more systematically.

sebbacon commented 4 years ago
brianmackenna commented 3 years ago

I'm signed up to quite a few alerts and the only one that consistently fails is the All-England alert. Prior to COVID I pestered Seb for maunal resend but not a priority during the epidemic. When we are spending time on OpenPrescribing it would be great if this could be prioritised for a fix as there are a certain select group of people who will be interested in national picture but not so much in signing up to a single practice or single CCG.

@sebbacon has suggested sending All-England alerts first above if this is potentially a quick fix on it without completely revisiting?

sebbacon commented 3 years ago

Here's a copy of the entire screen buffer from the last run

sebbacon commented 3 years ago

Type 1: bad email address, at openprescribing/common/alert_utils.py, line 40

Sending a message to sach.hirani@gp-c82038.nhs.uuk from OpenPrescribing <feedback@openprescribing.net>
Mailgun API response 400 (BAD REQUEST):
{
  "message": "'to' parameter is not a valid address. please check documentation"
}

Type 2: bad return from phantomjs HTTP activity

/usr/local/bin/phantomjs /webapps/openprescribing/openprescribing/frontend/management/commands/grab_chart.js "https://openprescribing
.net/analyse/#org=practice&orgIds=07R&numIds=4.1&denom=total_list_size&selectedTab=summary&source=button"

List of pages not returning:

https://openprescribing.net/analyse/#org=practice&orgIds=07R&numIds=4.1&denom=total_list_size&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=practice&orgIds=07R&numIds=4.1&denom=total_list_size&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=practice&orgIds=05T&numIds=0601023AQBB,0601023AQAA&denomIds=0601023ABAA,0601023ABBB&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=practice&orgIds=03T&numIds=0601023AQ&denomIds=0601023AB&selectedTab=map https://openprescribing.net/analyse/#org=practice&orgIds=P88026&numIds=21220000252&denom=nothing&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=practice&orgIds=07X&numIds=9&denom=nothing&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=pcn&orgIds=U87478&numIds=0407010Q0&denomIds=4.7&selectedTab=chart&source=button https://openprescribing.net/analyse/#org=pcn&orgIds=U21465&numIds=0403040ABAA&denom=total_list_size&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=CCG&numIds=21220000283&denom=nothing&selectedTab=summary&source=button https://openprescribing.net/analyse/#org=practice&orgIds=07R&numIds=4.1&denom=total_list_size&selectedTab=summary&source=button

sebbacon commented 3 years ago

Many of the links above are for closed CCGs, which return an error.

Others are timing out.

My quick fixes for today are:

  1. Move all england alerts to the start of the alert-sending process
  2. Remove all alerts for CCGs that don't exist any more
richiecroker commented 1 year ago

@inglesp @evansd is this still an issue?