CeleryEmailBackend ignores the fail_silently option when enqueuing messages in send_messages(). When the Celery message broker is down and DEBUG is False, this causes huge delays in requests where a server error occurs. This allows a DoS attack.
Steps to reproduce:
Ensure Celery is configured to retry failed publishing of tasks (on by default).
Set DEBUG to False in settings.py.
Stop the Celery message broker.
Send a request that causes a server error (you can simply raise an artificial error in a view and then call it).
The response takes about a minute (many times more than the expected time).
Analysis
A typical Django application has several middleware classes in front of the application views. The middleware classes are chained together. To propagate the request, each middleware calls get_response(), which translates to django.core.handlers.exception.convert_exception_to_response.inner(). Its code is:
When an error occurs, it is logged inside response_for_exception(). The default logging configuration attempts to send an error report to the website admins, and passes fail_silently to the email backend. But when the Celery message broker is down, CeleryEmailBackend does not suppress the exception thrown by Celery/Kombu (kombu.exceptions.OperationalError), and the exception escapes out of response_for_exception(). It is caught by the get_response() call of the previous middleware, which in turn repeats the process. This retry process is repeated as many times as the number of middleware classes configured, which significantly multiplies the expected response delay.
Note: This is not caused by a Celery issue ((https://github.com/celery/celery/issues/4296)). I am aware of this issue and have applied a workaround, but the problem described here still persists.
Solution
I have managed to work around this issue by wrapping CeleryEmailBackend in the following subclass:
logger = logging.getLogger(__name__)
class CustomCeleryEmailBackend(CeleryEmailBackend):
def send_messages(self, *args, **kwargs):
try:
super().send_messages(*args, **kwargs)
except OperationalError:
if not self.fail_silently:
raise
logger.exception('Error enqueuing messages.')
A fix in the library code would be similar and quite simple - wrap calls to send_emails.delay() in a try block, and if an OperationalError exception is caught and fail_silently is set, suppress the exception, possibly logging it to prevent it from being unnoticed.
Stacktrace
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/Users/victor/work/someapp/web/views/public.py", line 116, in index
raise Exception('test error')
Exception: test error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/work/someapp/web/middleware/apperror.py", line 10, in __call__
return self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/work/someapp/web/middleware/timezone.py", line 19, in middleware
return get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/blacklist/middleware.py", line 25, in middleware
return get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/work/someapp/web/middleware/proxy.py", line 82, in middleware
return get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/deprecation.py", line 94, in __call__
response = response or self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 42, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 141, in __call__
response = self.get_response(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/base.py", line 75, in get_response
response = self._middleware_chain(request)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/handlers/exception.py", line 95, in response_for_exception
exc_info=sys.exc_info(),
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 228, in log_response
exc_info=exc_info,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1412, in error
self._log(ERROR, msg, args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1519, in _log
self.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1529, in handle
self.callHandlers(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
hdlr.handle(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/logging/__init__.py", line 905, in handle
self.emit(record)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 122, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/utils/log.py", line 125, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
mail.send(fail_silently=fail_silently)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/django/core/mail/message.py", line 291, in send
return self.get_connection(fail_silently).send_messages([self])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/djcelery_email/backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
return self.apply_async(args, kwargs)
File "/Users/victor/work/someapp/web/tasks.py", line 10, in apply_async
return super().apply_async(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/task.py", line 570, in apply_async
**options
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/base.py", line 756, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 44, in __call__
value = self.__value__ = self.__contract__()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/messaging.py", line 224, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 852, in default_channel
self.ensure_connection(**conn_opts)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
return fun(*args, **kwargs)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/connection.py", line 307, in connect
self.transport.connect()
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/Users/victor/anaconda3/envs/someapp/lib/python3.7/site-packages/amqp/transport.py", line 161, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 61] Connection refused
[29/Jul/2019 15:10:12] "GET / HTTP/1.1" 500 59
Overview
CeleryEmailBackend
ignores thefail_silently
option when enqueuing messages insend_messages()
. When the Celery message broker is down andDEBUG
isFalse
, this causes huge delays in requests where a server error occurs. This allows a DoS attack.Steps to reproduce:
DEBUG
toFalse
insettings.py
.Expected behavior
The response should take 1-2 seconds, depending on the configured Celery message sending retry policy (see (https://docs.celeryproject.org/en/latest/userguide/calling.html#calling-retry)).
Actual behavior
The response takes about a minute (many times more than the expected time).
Analysis
A typical Django application has several middleware classes in front of the application views. The middleware classes are chained together. To propagate the request, each middleware calls
get_response()
, which translates todjango.core.handlers.exception.convert_exception_to_response.inner()
. Its code is:When an error occurs, it is logged inside
response_for_exception()
. The default logging configuration attempts to send an error report to the website admins, and passesfail_silently
to the email backend. But when the Celery message broker is down,CeleryEmailBackend
does not suppress the exception thrown by Celery/Kombu (kombu.exceptions.OperationalError
), and the exception escapes out ofresponse_for_exception()
. It is caught by theget_response()
call of the previous middleware, which in turn repeats the process. This retry process is repeated as many times as the number of middleware classes configured, which significantly multiplies the expected response delay.Note: This is not caused by a Celery issue ((https://github.com/celery/celery/issues/4296)). I am aware of this issue and have applied a workaround, but the problem described here still persists.
Solution
I have managed to work around this issue by wrapping
CeleryEmailBackend
in the following subclass:A fix in the library code would be similar and quite simple - wrap calls to
send_emails.delay()
in a try block, and if anOperationalError
exception is caught andfail_silently
is set, suppress the exception, possibly logging it to prevent it from being unnoticed.Stacktrace