search

sibson / redbeat

RedBeat is a Celery Beat Scheduler that stores the scheduled tasks and runtime metadata in Redis.
Apache License 2.0
916 stars 130 forks source link

Redbeat doesn't work with sentinel #211

Closed HungPhann closed 1 year ago

HungPhann commented 3 years ago

I'm using redbeat==2.0.0. It works fine with normal Redis but when being configured for sentinel, it got some issues. This is my config for sentinel:

   redbeat_redis_url = f"redis-sentinel://redis-sentinel:26379/{cfg.REDIS['db']}"
   redbeat_redis_options = {
        'sentinels': [(cfg.REDIS['host'], cfg.REDIS['port'])],
        'password': cfg.REDIS['password'],
        'db': {cfg.REDIS['db']},
        'service_name': cfg.REDIS["master"],
        'socket_timeout': cfg.REDIS["timeout"],
        'retry_period': 180
    }

When starting beat, I got this error:

celery beat v4.4.7 (cliffs) is starting.
__    -    ... __   -        _
LocalTime -> 2021-10-14 03:24:16
Configuration ->
    . broker -> sentinel://:**@redis-sentinel:26379/3
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> redbeat.schedulers.RedBeatScheduler
       . redis -> redis-sentinel://redis-sentinel:26379/3
       . lock -> `goldfinger_staging` 30.00 seconds (30s)
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
2021-10-14 03:24:16,629 - celery.beat -  - - DEBUG - Setting default socket timeout to 30
2021-10-14 03:24:16,630 - celery.beat -  - - INFO - beat: Starting...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/objects.py", line 42, in __get__
2021-10-14 03:24:16,635 - celery.beat -  - - CRITICAL - beat raised exception <class 'redis.exceptions.DataError'>: DataError("Invalid input of type: 'set'. Convert to a bytes, string, int or float first.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/apps/beat.py", line 109, in start_scheduler
    service.start()
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 622, in start
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    humanize_seconds(self.scheduler.max_interval))
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 666, in scheduler
    return self.get_scheduler()
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 657, in get_scheduler
    return symbol_by_name(self.scheduler_cls, aliases=aliases)(
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 377, in __init__
    super(RedBeatScheduler, self).__init__(app, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 257, in __init__
    self.setup_schedule()
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 382, in setup_schedule
    previous = set(key for key in client.smembers(self.app.redbeat_conf.statics_key))
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 16, in main
    return self(f, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
    _main()
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 322, in main
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 89, in retrier
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 2281, in smembers
    return self.execute_command('SMEMBERS', name)
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.9/site-packages/redis/sentinel.py", line 44, in connect
    self.connect_to(self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.9/site-packages/redis/sentinel.py", line 34, in connect_to
    super(SentinelManagedConnection, self).connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 567, in connect
    self.on_connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 663, in on_connect
    self.send_command('SELECT', self.db)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 775, in pack_command
    for arg in imap(self.encoder.encode, args):
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 119, in encode
    raise DataError("Invalid input of type: '%s'. Convert to a "
redis.exceptions.DataError: Invalid input of type: 'set'. Convert to a bytes, string, int or float first.
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 491, in handle_argv
    return self.execute(command, argv)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 415, in execute
    return cls(
  File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 308, in run_from_argv
    return self.handle_argv(prog_name,
  File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 393, in handle_argv
    return self(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 253, in __call__
    ret = self.run(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/beat.py", line 109, in run
    return beat().run()
  File "/usr/local/lib/python3.9/site-packages/celery/apps/beat.py", line 81, in run
    self.start_scheduler()
  File "/usr/local/lib/python3.9/site-packages/celery/apps/beat.py", line 109, in start_scheduler
    service.start()
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 622, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 666, in scheduler
    return self.get_scheduler()
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 657, in get_scheduler
    return symbol_by_name(self.scheduler_cls, aliases=aliases)(
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 377, in __init__
    super(RedBeatScheduler, self).__init__(app, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/beat.py", line 257, in __init__
    self.setup_schedule()
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 382, in setup_schedule
    previous = set(key for key in client.smembers(self.app.redbeat_conf.statics_key))
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redbeat/schedulers.py", line 89, in retrier
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 2281, in smembers
    return self.execute_command('SMEMBERS', name)
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.9/site-packages/redis/sentinel.py", line 44, in connect
    self.connect_to(self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.9/site-packages/redis/sentinel.py", line 34, in connect_to
    super(SentinelManagedConnection, self).connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 567, in connect
    self.on_connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 663, in on_connect
    self.send_command('SELECT', self.db)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 775, in pack_command
    for arg in imap(self.encoder.encode, args):
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 119, in encode
    raise DataError("Invalid input of type: '%s'. Convert to a "
redis.exceptions.DataError: Invalid input of type: 'set'. Convert to a bytes, string, int or float first.
nigel-gott commented 2 years ago

Are you sure your config is correct?

It looks like you are setting a python set literal on redbeat_redis_options['db'] and hence the error you are seeing:

'db': {cfg.REDIS['db']}, This is a set literal and not a string!

it should instead be 'db': cfg.REDIS['db'],

sibson commented 1 year ago

If you are still experiencing issues, please reopen the issue and provide additional details to help debug further.