search

milesmcc / shynet

Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.
Apache License 2.0
2.87k stars 180 forks source link

No support for redis sentinel caches. #285

Open kashalls opened 10 months ago

kashalls commented 10 months ago

It seems like the redis package does not have support for redis sentinel databases. Redis sentinel allows for better high availability support than running a single redis instance. 


Launching Shynet queue worker...

 -------------- celery@shynet-celeryworker-84bc85849d-xqb7c v5.2.7 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-6.1.0-9-amd64-x86_64-with 2023-09-19 09:36:44
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         shynet:0x7f9ba37facb0
- ** ---------- .> transport:   redis://redis.default.svc.cluster.local:6379/6
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ---- .> task events: ON
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

[tasks]
  . analytics.tasks.ingress_request
  . dashboard.tasks.send_email

[2023-09-19 09:36:44,579: INFO/MainProcess] Connected to redis://redis.default.svc.cluster.local:6379/6
[2023-09-19 09:36:44,584: INFO/MainProcess] mingle: searching for neighbors
[2023-09-19 09:36:45,602: INFO/MainProcess] mingle: all alone
[2023-09-19 09:36:45,624: CRITICAL/MainProcess] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/gossip.py", line 107, in start
    super().start(c)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 397, in start
    self.consumers = self.get_consumers(channel)
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/gossip.py", line 175, in get_consumers
    return [Consumer(
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 387, in __init__
    self.revive(self.channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 409, in revive
    self.declare()
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 422, in declare
    queue.declare()
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
    self._create_queue(nowait=nowait, channel=channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 617, in _create_queue
    self.queue_bind(nowait=nowait, channel=channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 660, in queue_bind
    return self.bind_to(self.exchange, self.routing_key,
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 669, in bind_to
    return (channel or self.channel).queue_bind(
  File "/usr/local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 562, in queue_bind
    self._queue_bind(exchange, *meta)
  File "/usr/local/lib/python3.10/site-packages/kombu/transport/redis.py", line 1033, in _queue_bind
    client.sadd(self.keyprefix_queue % (exchange,),
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 2243, in sadd
    return self.execute_command('SADD', name, *values)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.`
milesmcc commented 10 months ago

@Kashalls thanks for the report. Are we good to close this issue now that #287 is merged in?

Btw, if you'd like to submit a PR documenting Sentinel cache support (following the change in #287) that would be quite helpful (as right now Shynet users may not know that support exists without finding this issue/reading the code).

kashalls commented 10 months ago

Btw, if you'd like to submit a PR documenting Sentinel cache support (following the change in #287) that would be quite helpful (as right now Shynet users may not know that support exists without finding this issue/reading the code).

I'll work on it this weekend, thanks for reminding me.

kashalls commented 10 months ago

Okay so it looks like the issue is a lot bigger than I initially thought it would be.

I didn't notice until today that this happened on the frontend side:

Performing startup checks...
Database is ready to go.
Startup checks complete!
Launching Shynet web server...
[2023-09-24 22:11:16 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-09-24 22:11:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2023-09-24 22:11:16 +0000] [1] [INFO] Using worker: sync
[2023-09-24 22:11:16 +0000] [9] [INFO] Booting worker with pid: 9
ERROR Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 92, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 146, in dispatch
    return super(LoginView, self).dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 74, in dispatch
    response = super(RedirectAuthenticatedUserMixin, self).dispatch(
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 101, in post
    if form.is_valid():
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 438, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 459, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.10/site-packages/allauth/account/forms.py", line 177, in clean
    user = get_adapter(self.request).authenticate(self.request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 511, in authenticate
    self._delete_login_attempts_cached_email(request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 483, in _delete_login_attempts_cached_email
    cache.delete(cache_key)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 30, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 287, in delete
    return client.delete(key)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 1567, in delete
    return self.execute_command('DEL', *names)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.
Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 92, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 146, in dispatch
    return super(LoginView, self).dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 74, in dispatch
    response = super(RedirectAuthenticatedUserMixin, self).dispatch(
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 101, in post
    if form.is_valid():
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 438, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 459, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.10/site-packages/allauth/account/forms.py", line 177, in clean
    user = get_adapter(self.request).authenticate(self.request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 511, in authenticate
    self._delete_login_attempts_cached_email(request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 483, in _delete_login_attempts_cached_email
    cache.delete(cache_key)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 30, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 287, in delete
    return client.delete(key)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 1567, in delete
    return self.execute_command('DEL', *names)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

So it seems DJANGO needs to be fixed to allow support for it.

The pr #287 did not fix it at all because even though json.loads can import the json from the environment variable, I think it converts it into a dict class and even though you can have nested dict's it still complains about ValueError: dictionary update sequence element #0 has length 1; 2 is required in celery when combining the default options.

Basically the options in settings.py needs to be set like this: CELERY_BROKER_TRANSPORT_OPTIONS = { 'master_name': 'redis-master' } and not: CELERY_BROKER_TRANSPORT_OPTIONS: "{'master_name': 'redis-master'}"

kashalls commented 10 months ago

This is starting to become outside of my knowledge, and will likely require outside help to patch this.