Growth team would love for us to be able to give users <30 sec feedback loop on ingestion during onboarding.
For onboarding, ingestion delays can really impact the new user experience and be super frustrating.
Options to consider
have information about the lag
a problem here is that the lag if it's low can depend on which partition the message ends up on so the lag will vary between partitions, and showing the max of those might make us look worse.
have a priority queue for new orgs
that seems quite complex to identify who should be there, when to move them around in addition to having a separate queue which is complex
build live tailing (Xavier has mentioned this idea in the past)
potentially ad-hoc spin-up a Kafka consumer that gives us messages as they come from capture
Separate basic events and person processing, which would allow basic events to be processed much faster
we could have basic events table in CH, that has a short TTL
pipeline would write to that table, and person processing happens separately taking events from there and adding them to the real events table with person_id and person_properties
Growth team would love for us to be able to give users <30 sec feedback loop on ingestion during onboarding.
For onboarding, ingestion delays can really impact the new user experience and be super frustrating. Options to consider