Plugin request: Snowflake

PostHog / plugin-repository

Plugins for PostHog

MIT License

2 stars 7 forks source link

Plugin request: Snowflake #13

Closed timgl closed 3 years ago

timgl commented 3 years ago

I want to dump all my events and person information in Snowflake.

~I've actually had a look at this. it seems the only way to do this is to create .csv's of events, upload them to s3 and then make a call to snowflake to ingest that. I'm trying to get a direct contact there to ask if there's an easier way, and to get added to their PartnerConnect program.~

I think the easiest way to do this is to use their node connector, then do something like this to insert it one at a time.

mariusandra commented 3 years ago

Depends on https://github.com/PostHog/plugin-repository/issues/12 then

jamesefhawkins commented 3 years ago

I've had no response from the contact I got referred to, I've a third person I've been told I can get connected to through an investor (support = world networking championships, clearly), have asked him to intro! I've sent this question directly.

mariusandra commented 3 years ago

This is built: https://github.com/PostHog/snowflake-export-plugin

Some points:

The following needs to be merged for it to work: https://github.com/PostHog/plugin-server/pull/291
Snowflake supports a few different methods of authentication. Currently only "password" is implemented. Adding the "private key" option shouldn't be that much extra work.
Finally, we have the same case as with the bigquery export: there's no retry support with this plugin. In case snowflake or your network is down at some point, the events are lost. Good enough for now, but we need to find a universal solution to it (see https://github.com/PostHog/plugin-server/issues/273)

timgl commented 3 years ago

I think we can safely ignore $snapshot and $feature_flag_called events for now (and maybe even autocapture). Other than that great work!

mariusandra commented 3 years ago

I can add some option in the plugin config to select what types of events to send. Who knows, someone might actually want to send their $snapshots somewhere for archival. The BigQuery plugin theoretically suffers from the same situation as it also just sends everything.

mariusandra commented 3 years ago

I added this (default values in screenshot):

timgl commented 3 years ago

yeah i think that's fine for now. would be nice to standardise that somehow but not urgent

Twixes commented 3 years ago

Snowflake contact woes aside, I would say this can be considered done #18