search

coleifer / huey

a little task queue for python
https://huey.readthedocs.io/
MIT License
5.18k stars 370 forks source link

Error with huey.SqliteHuey #696

Closed kravciuk closed 1 year ago

kravciuk commented 1 year ago

Django==4.1.4 , Python 3.10

Setting part:

`HUEY = {
    'huey_class': 'huey.SqliteHuey',  # Huey implementation to use.
    'name': settings.DATABASES['default']['NAME'],  # Use db name for huey.
    'results': True,  # Store return values of tasks.
    'store_none': False,  # If a task returns None, do not save to results.
    'immediate': settings.DEBUG,  # If DEBUG=True, run synchronously.
    'utc': True,  # Use UTC for all times internally.
    'blocking': True,  # Perform blocking pop rather than poll Redis.
    'connection': {
        'host': 'localhost',
        'port': 6379,
        'db': 0,
        'connection_pool': None,  # Definitely you should use pooling!
        # ... tons of other options, see redis-py for details.

        # huey-specific connection parameters.
        'read_timeout': 1,  # If not polling (blocking pop), use timeout.
        'url': None,  # Allow Redis config via a DSN.
    },
    'consumer': {
        'workers': 1,
        'worker_type': 'thread',
        'initial_delay': 0.1,  # Smallest polling interval, same as -d.
        'backoff': 1.15,  # Exponential backoff using this rate, -b.
        'max_delay': 10.0,  # Max possible polling interval, -m.
        'scheduler_interval': 1,  # Check schedule every second, -s.
        'periodic': True,  # Enable crontab feature.
        'check_worker_health': True,  # Enable worker health checks.
        'health_check_interval': 1,  # Check worker health every second.
    },
}

And error after project start:

Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/home/vadim/.pyenv/versions/3.10.4/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/home/vadim/.pyenv/versions/3.10.4/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/www/dev/.venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/www/dev/.venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 125, in inner_run
    autoreload.raise_last_exception()
  File "/www/dev/.venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 87, in raise_last_exception
    raise _exception[1]
  File "/www/dev/.venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 398, in execute
    autoreload.check_errors(django.setup)()
  File "/www/dev/.venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/www/dev/.venv/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/www/dev/.venv/lib/python3.10/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/www/dev/.venv/lib/python3.10/site-packages/django/apps/config.py", line 193, in create
    import_module(entry)
  File "/home/vadim/.pyenv/versions/3.10.4/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/contrib/djhuey/__init__.py", line 100, in <module>
    HUEY = backend_cls(name, **huey_config)
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/api.py", line 116, in __init__
    self.storage = self.create_storage()
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/api.py", line 135, in create_storage
    return self.get_storage(**self.storage_kwargs)
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/api.py", line 147, in get_storage
    return Storage(self.name, **kwargs)
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 713, in __init__
    super(SqliteStorage, self).__init__(name)
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 628, in __init__
    self.initialize_schema()
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 664, in initialize_schema
    with self.db(commit=True, close=True) as curs:
  File "/home/vadim/.pyenv/versions/3.10.4/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 647, in db
    conn = self.conn
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 639, in conn
    self._state.set_connection(self._create_connection())
  File "/www/dev/.venv/lib/python3.10/site-packages/huey/storage.py", line 716, in _create_connection
    conn = sqlite3.connect(self.filename, timeout=self._timeout,
TypeError: function takes at most 8 arguments (9 given)
coleifer commented 1 year ago

I think the issue is you are passing redis specific parameters to the storage initializer:

'connection': {
        'host': 'localhost',
        'port': 6379,
        'db': 0,
        'connection_pool': None,  # Definitely you should use pooling!
        # ... tons of other options, see redis-py for details.

        # huey-specific connection parameters.
        'read_timeout': 1,  # If not polling (blocking pop), use timeout.
        'url': None,  # Allow Redis config via a DSN.
    },

For SQLite you should remove all that.

coleifer commented 1 year ago

Here are the supported parameters for the SQLite storage in case you're curious: https://github.com/coleifer/huey/blob/master/huey/storage.py#L692