Open macobo opened 2 years ago
This makes sense to me, my question though is, don't we plan to include persons on events as part of… persons-on-events anyway? In which case the rearranging of the pipeline still makes sense, but it would come more naturally.
The two projects are independent as it's the "2. "Prepare" events for ingestion (We fetch/update/create persons/groups here)" that is enriching events with person information currently. Note this happens after we run the plugins.
Yakko and I have discussed the buffer part a bit and here's the related comment: https://github.com/PostHog/posthog/issues/9182#issuecomment-1134516822 so combining that with the proposal here the pipeline in the future could be:
Once https://github.com/PostHog/posthog/pull/10360 lands there's not much left to do:
Undo radical change to event pipeline - will re-add it later!
.Note we should wait for https://github.com/PostHog/posthog/pull/10355/files before that change.
Note that this becomes higher priority once we start rate limiting the api per #12015 because currently several teams use calls to /persons
to sync person data to a warehouse/other destination. Once we have this info in the plugin server, the export plugins can be updated to pass person info along. There are at least two teams I know of (discussed here ) that this will affect, but probably more.
Is your feature request related to a problem?
Currently, person (and group) information is not exposed to plugins. There are several plugins which work around this by consistently hitting the API.
Describe the solution you'd like
Expose this information in the plugin events payload. This involves reworking the event pipeline (IMO in a good way).
Current state of event pipeline
The current event pipeline looks like this:
Note that the step 2 is the "fattest" currently and is intermingled with event handling logic. Cleaning this up will clean up our logic somewhat.
We also do lookups in step 3 though we create persons in step 2, creating possibility for simplifications
Proposed event pipeline
We might not want to expose group information to plugins themselves in the first stage and only expose person information. This is a product decision which might simplify refactoring in the future.
Note that we pass forward person objects from one step to the next here.
Problems and workarounds
Additional context
cc @yakkomajuri @tiina303 @Twixes
I think this is both a win for our users as well as cleans up the plugin-server/src/worker/ingestion/process-event.ts file which is doing perhaps too much.
It also is something that's easier to implement before the buffer is live - changing the payloads afterwards will cause trouble and extra debugging.
Thank you for your feature request – we love each and every one!