search

danthegoodman1 / OpenSigma

Open source version of Stripe Sigma - Stripe event archiving and analytics without fees
1 stars 0 forks source link

Make storage interface #14

Closed danthegoodman1 closed 1 year ago

danthegoodman1 commented 1 year ago

Generic interface so multiple storage providers can be used.

Should have:

  1. Init (connect, create tables and MVs)
  2. Insert events

Also should provide any migrations or files needed to setup. They can expect their own env vars.

Can start by making clickhouse and/or tinybird initially, maybe add icedb later

danthegoodman1 commented 1 year ago

For CH and tinybird want:

  1. Raw events (time)
  2. Event,time?
  3. Latest event for ID (argMax, max)
  4. Events by email (changed order by)
  5. Banned Signals (email, address, etc.)
danthegoodman1 commented 1 year ago

Need to figure out inverted indexing for emails so we can find related addresses and such to consider default banned when they get banned.