search

PostHog / posthog

🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.
https://posthog.com
Other
19.31k stars 1.13k forks source link

Messaging queue #14385

Open yanivbh1 opened 1 year ago

yanivbh1 commented 1 year ago

First of all, we are using and enjoying PostHog. It's really a helpful product!

In what situation are you experiencing subpar performance?

Events ingestion and export using 3rd party integrations

Please describe. Recently I noticed a few performance issues as our usage grew, and after digging up a bit in the open-source repo, I found that you are using RabbitMQ and Celery. As PostHog continues to grow rapidly, I assume the need for more scale, multi-tenancy, and priority queues has risen. Therefor, I thought thats the perfect time to bring up Memphis.dev, which can help you get there in no-time, and with minor dev efforts, resources, and adaptations.

Minor note, we are also working on integrating Memphis within PostHog Apps.

How to reproduce

No specific action has made.

Environment

Additional context

Thank you for your performance issue report – we want PostHog to go supersonic!

weyert commented 1 year ago

I thought it was using Kafka?

yanivbh1 commented 1 year ago

I thought it was using Kafka?

PostHog?

neilkakkar commented 1 year ago

Yep, we use Kafka for our ingestion queue; the celery queue is for internal tasks (like cohort recalculation).

Mind elaborating a bit more about what kind of perf issues you are facing?