Heroku hosted Posthog fails with SSL errors after Redis 7 upgrade

rrosztoczy commented 1 year ago

Bug description

Please describe.
Our posthog application (hosted in Heroku via one click setup using gunicorn posthog.wsgi) started failing do to SSL errors after a forced Redis upgrade by Heroku to Redis 7.0.11 (who no longer supports anything below 7). I found a discussion in the celery repo with the same issue. It looks like they needed to update the version of Redis in their requirements.txt from 4.5.4 to 4.4.4 which solved the issue and I am wondering if it would be the same resolution here. Error log:


 Internal Server Error: /engage/
 Traceback (most recent call last):
   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/utils/objects.py", line 42, in __get__
     return obj.__dict__[self.__name__]
 KeyError: 'backend'

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
     response = get_response(request)
   File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File "/app/.heroku/python/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
     return view_func(*args, **kwargs)
   File "/app/posthog/api/capture.py", line 226, in get_event
     capture_internal(event, distinct_id, ip, site_url, now, sent_at, team.pk, event_uuid)  # type: ignore
   File "/app/posthog/api/capture.py", line 282, in capture_internal
     celery_app.send_task(
   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/base.py", line 774, in send_task
     self.backend.on_task_call(P, task_id)
   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/utils/objects.py", line 44, in __get__
     value = obj.__dict__[self.__name__] = self.__get(obj)
   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/base.py", line 1227, in backend
     return self._get_backend()
   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/base.py", line 945, in _get_backend
     return backend(app=self, url=url)
   File "/app/.heroku/python/lib/python3.8/site-packages/celery/backends/redis.py", line 269, in __init__
     raise ValueError(E_REDIS_SSL_CERT_REQS_MISSING_INVALID)
 ValueError: 
 A rediss:// URL must have parameter ssl_cert_reqs and this must be set to CERT_REQUIRED, CERT_OPTIONAL, or CERT_NONE

While the issues point to the ssl_cert_reqs parameter, I believe that is misleading as going that route only led to additional errors (and supposedly Heroku manages that). Additionally, the change in required version in the celery discussion resolved the issue. These issues come from the core posthog app gunicorn posthog.wsgi not the celery worker (which we do not run).

Link to discussion:

https://github.com/celery/celery/discussions/8334

How to reproduce

Any application hosted on heroku should be encountering the same issue

Environment

[X] Hosted with one click heroku setup
[ ] PostHog Cloud US, project ID: [please provide from https://app.posthog.com/project/settings#project-id]
[ ] PostHog Cloud EU, project ID: [please provide from https://eu.posthog.com/project/settings#project-id]
[ ] PostHog Hobby self-hosted with docker compose, version/commit: [please provide]
[ ] PostHog self-hosted with Kubernetes (deprecated, see "Sunsetting Kubernetes support"), version/commit: [please provide]

Additional context

Thank you for your bug report – we love squashing them!

spakanati commented 1 year ago

We're having the same issue and would love to know if there is a fix!

rrosztoczy commented 1 year ago

My preferred solution would be to migrate to post hog cloud, but we need to be able to get our old servers up in order to do that. Still waiting on an answer from the PH team!

PostHog / posthog