Design Document

[ats1999]

Work Flow

1. The user will install our application. At the time of installation, the user can supply a password for the WatchMan user. The WatchMan user will be created by the WatchMan server startup. If the user doesn’t exit.
2. The WatchMan user can create any new user with the help of API. WatchMan will supply a username and password while creating any new user.
3. Any user, including WatchMan can change his/her password.
4. Any user, including WatchMan can create events, schema, rules, etc.
5. Any consumer can send an event to the system by supplying an event-name, payload, configurations(rate-limit, sampling, etc) and API key for the event.
   6. API key can be read only or write only
   7. Name them publisher key and analyst key
6. The Event Will be pushed into Kafka
7. Kafka consumers will poll events validate their schema, and discard invalid events. Push valid events into another kafka topic to be pushed into ClickHouse.
8. As of now data has been pushed into click house, any user with valid API key can now query data.

Schema

User

user_id:string
password:string

Event

event_id:string // unique
user_id:string
event_name:string 
sampling_percentage:number // sampling percentage

// SERVER | CLIENT | QUERY
// https://github.com/ats1999/WatchMan/issues/6
sampling_type:string | number(1,2,3, .. id of the type)

data_deletion_policy: number // amount of milliseconds
reporting_time_zone: PST // keep a enum type

Event Log

event_id:string
user_id:string
time_stamp:number
log:object
   - country
   - device

Event API Key

user_id:string
event_id:string
key_name:string // unique identifier
key_type:string // publisher | analyst

[ats1999]

https://clickhouse.com/docs/en/sql-reference/data-types/map

[ats1999]

https://clickhouse.com/docs/en/sql-reference/data-types

[ats1999]

Approaches to handle user-defined schema

• Using new table for every event
• Storing values as Map data type
• Storing values as JSON data type
• Have a limit on a number of dimensions per event, then use some pre-defined random column names to store and query data.