consumer handlers will not load

[acheraime]

I've followed all the instructions but the consumer handler will not work, all the configuration are being ignored. See below celery.py and consumer configuration files.

celery.py

from celery import Celery
from event_consumer.handlers import AMQPRetryConsumerStep

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<proj>.settings')
os.environ.setdefault('EVENT_CONSUMER_APP_CONFIG', '<proj>.consettings')

app = Celery()

app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

consumer_app = Celery()
consumer_app.config_from_envvar('EVENT_CONSUMER_APP_CONFIG')

consumer_app.steps['consumer'].add(AMQPRetryConsumerStep)

config file

EVENT_CONSUMER_USE_DJANGO = True

EVENT_CONSUMER_EXCHANGES = {
    # a reference name for this config, used when attaching handlers
    'default': {
        'name': 'TAP.Events',  # actual name of exchange in RabbitMQ
        'type': 'topic',  # an AMQP exchange type
    }
}

CELERY_IMPORTS = ['system.consumers', ]

[anentropic]

are you relying on setdefault to set the value of EVENT_CONSUMER_APP_CONFIG?

if so you might need to move that line before the from event_consumer import

[acheraime]

that did not fix it.

[acheraime]

can you share a working celery.py example?

[anentropic]

have you tried actually setting the env var in your environment, rather than rely on setdefault to update os.environ dict?

how do you know it's "not working"?

are you starting your worker with -A celery:consumer_app?

[acheraime]

I did all that still no success. I know because celery log does not show any message related to my consumer. All my other celery tasks work.

[acheraime]

When I move the os.environ line before "from event_consumer import ..." I got below error

File "/usr/share/virtualenvs/connect/local/lib/python2.7/site-packages/flexisettings/__init__.py", line 61, in __init__
    config = _load_config(initial_namespace, defaults)
  File "/usr/share/virtualenvs/connect/local/lib/python2.7/site-packages/flexisettings/__init__.py", line 111, in _load_config
    _temp.update_from_object(app_config, lambda key: key.startswith(namespace))
  File "/usr/share/virtualenvs/connect/local/lib/python2.7/site-packages/configloader/__init__.py", line 66, in update_from_object
    mod = __import__(path, globals(), locals(), [name], 0)
TypeError: Item in ``from list'' not a string

[anentropic]

@acheraime this seems important

whatever value you have set for EVENT_CONSUMER_APP_CONFIG it seems like that file can't be imported

[lotaku]

I had a similar problem yesterday.

You need to set the environment variables first.

os.environ.setdefault('EVENT_CONSUMER_APP_CONFIG', str('xxx.csm_settings'))
os.environ.setdefault('EVENT_CONSUMER_CONFIG_NAMESPACE', str('EVENT_CONSUMER'))

Then perform the import.

from event_consumer.handlers import AMQPRetryConsumerStep
consumer_app.config_from_object('django.conf:settings')
consumer_app.steps['consumer'].add(AMQPRetryConsumerStep)

Problem solved.