Non-default connection and database location settings are not used

[P-T-I]

Even though I added the correct keys into my celery app.conf, celerybeat-mongo does not find / uses them and falls back to the default collection, db and host-uri.

app.conf.update(
    CELERY_MONGODB_SCHEDULER_DB="database",
    CELERY_MONGODB_SCHEDULER_COLLECTION="schedules",
    CELERY_MONGODB_SCHEDULER_URL="mongodb://127.0.0.1:27017"
)

For whatever reason the current_app.conf does not hold the keys above and the defaults are initialized. I confirmed that the above config is send correctly to the celery app, I can manually retrieve the settings and the values match as they where set in the app.conf.update above.

With some minor alterations to the MongoScheduler init method (close to the same changes @rafaelreuber suggests to the MongoScheduler instantiation) I was able to succesfully push non-default settings towards celerybeat-mongo.

[rafaelreuber]

You need to pass you celery app for the celerybeatmongo.schedulers.MongoScheduler class instantiation:

from celery import Celery
app = Celery('hello', broker='redis://localhost//')
app.conf.update({
    "mongodb_scheduler_db": "test_database",
    "mongodb_scheduler_url": "mongodb://localhost:27017",
})

from celerybeatmongo.schedulers import MongoScheduler
scheduler = MongoScheduler(app=app)

[P-T-I]

I agree it would, however my requirement is running the Celery daemon as a linux service; so I cannot pass that reference. However there is a app reference already in the celery.beat.Schedular class, that MongoScheduler inherits. So bumping the super init call further up the method makes it available. I've created a fork with a proposed fix.

[rafaelreuber]

When you run the command celery beat -A proj -S celerybeatmongo.schedulers.MongoScheduler, the celery beat command uses the value of the argument -A to pass the celery instance from you application to the MongoSheduler class.

Can you share a sample that reproduce your problem?

[P-T-I]

Yes I will do that as soon as I get the chance; have a busy weekend ahead so probably I’ll do that on Monday!

Op 15 mei 2020 om 23:24 heeft Rafael Reuber notifications@github.com het volgende geschreven:

 When you run the command celery beat -A proj -S celerybeatmongo.schedulers.MongoScheduler, the celery beat command uses the value of the argument -A to pass the celery instance from you application to the MongoSheduler class.

Can you share a sample that reproduce your problem?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[rafaelreuber]

No problem. Stay safe @P-T-I

[P-T-I]

@rafaelreuber

My code is fairly straight forward:

MyApp:

app = Celery('TheApp',
             broker='pyamqp://******:*********@localhost:5672/',
             backend='rpc://',
             include=['Myapp.tasks.test'])

app.conf.update(
    CELERYD_TASK_SOFT_TIME_LIMIT=3600,
    CELERYD_TASK_TIME_LIMIT=4800,
    CELERY_MONGODB_SCHEDULER_DB="custom_database",
    CELERY_MONGODB_SCHEDULER_COLLECTION="schedules",
    CELERY_MONGODB_SCHEDULER_URL="mongodb://127.0.0.1:27017"
)

@app.task(ignore_result=True)
def test():

    time.sleep(10)

    return 'Test'

beat.service:

[Unit]
Description=Celery Beat Service
After=network.target

[Service]
Type=simple
EnvironmentFile=/etc/default/Myapp
WorkingDirectory=/path/to/app/
ExecStart=/bin/sh -c '${CELERY_BIN} beat  \
  -A ${CELERY_APP} -S ${BEAT_SCHEDULER} --pidfile=${CELERYBEAT_PID_FILE} \
  --logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} '

[Install]
WantedBy=multi-user.target

Variables are loaded through a config file, I leave out most of the variables, just the ones that matter... /etc/deafult/Myapp:

[SNIP]
CELERY_APP="Myapp"
CELERYD_LOG_FILE="/var/log/Myapp/Myapp.log"
CELERYD_PID_FILE="/var/run/Myapp/%n.pid"
BEAT_SCHEDULER="celerybeatmongo.schedulers.MongoScheduler"
[SNIP]

[P-T-I]

@rafaelreuber, any chance on reproducing?

[rafaelreuber]

I noticed you're running celery beat as an systemd service. This problem just happens in this case?

It works as expected when you run celery beat through terminal as follow?

celery -A Myapp beat -l DEBUG -S celerybeatmongo.schedulers.MongoScheduler

[Jean-PhilippeD]

Did MongoScheduler became MongoPersistentScheduler ?

As I was affected by this issue, I updated with the latest release, but I can not start anymore celery beat with the command:

celery -A Myapp beat -l DEBUG -S celerybeatmongo.schedulers.MongoScheduler

I tried :

celery -A Myapp beat -l DEBUG -S celerybeatmongo.schedulers.MongoPersistentScheduler

But I got kind of auth error: command count requires authentication While I did not changed any settings.

[P-T-I]

Rafael,

Sorry, been away for a while; just tested it, there is no difference between systemd and command line activation, same results....

——————————

Paul

Op 29 mei 2020 om 23:50 heeft Rafael Reuber notifications@github.com het volgende geschreven:

 I noticed you're running celery beat as an systemd service. This problem just happens in this case?

It works as expected when you run celery beat through terminal as follow?

celery -A Myapp beat -l DEBUG -S celerybeatmongo.schedulers.MongoScheduler — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[rafaelreuber]

Hy @P-T-I

I added some debug log on this fork.

Please, install it using the following command:

pip install https://github.com/rafaelreuber/celerybeat-mongo/archive/debug_config.zip

Execute celerybeatmongo with DEBUG log level:

celery -A Myapp beat -l DEBUG -S celerybeatmongo.schedulers.MongoScheduler

Paste the output here.

[P-T-I]

Seems to do the trick; tried it for both the systemd service as well as running from the cli

Output:

LocalTime -> 2020-06-12 04:56:45
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celerybeatmongo.schedulers.MongoScheduler

    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 seconds (5s)
2020-06-12 04:56:45,588 - celery.beat - DEBUG    - Setting default socket timeout to 30
2020-06-12 04:56:45,589 - celery.beat - INFO     - beat: Starting...
2020-06-12 04:56:45,589 - celerybeatmongo.schedulers - DEBUG    - mongodb_scheduler_db: test
2020-06-12 04:56:45,590 - celerybeatmongo.schedulers - DEBUG    - mongodb_scheduler_connection_alias: default
2020-06-12 04:56:45,590 - celerybeatmongo.schedulers - DEBUG    - mongodb_scheduler_url: mongodb://127.0.0.1:27017
2020-06-12 04:56:45,590 - celerybeatmongo.schedulers - INFO     - backend scheduler using mongodb://127.0.0.1:27017/test:schedules
2020-06-12 04:56:45,590 - celery.beat - DEBUG    - beat: Ticking with max interval->5.00 seconds
2020-06-12 04:56:45,590 - celerybeatmongo.schedulers - DEBUG    - Writing entries...
2020-06-12 04:56:45,591 - celery.beat - DEBUG    - beat: Waking up in 5.00 seconds.
2020-06-12 04:56:46,470 - celerybeatmongo.schedulers - DEBUG    - Writing entries...
2020-06-12 04:56:46,471 - celerybeatmongo.schedulers - DEBUG    - Writing entries...

[Jean-PhilippeD]

Hey @rafaelreuber

I also installed your archive and it solved the problem !

But when I install the one one pypi repo, I have the problem I described in my last post.

[rafaelreuber]

I sent some PR to this repository. I'm waiting @zakird review them.

[P-T-I]

Awesome!

Op 12 jun. 2020 om 22:31 heeft Rafael Reuber notifications@github.com het volgende geschreven:

 I sent some PR to this repository. I'm waiting @zakird review them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.