Scope has an optional multitenant deployment model:
1 or more "collector" processes save reports to store (e.g. S3+DynamoDB).
Probes send reports to a load-balancer front-end which distributes incoming reports across collectors.
1 or more "query" processes handle requests from browsers, fetch reports from store, merge and render them.
Currently, the collectors save all reports to store in near-real-time, and queriers do all rendering by querying the store as-at the time required. Live rendering is done by repeatedly querying as-at "now".
New proposal, which makes the store optional for live rendering:
Each collector process holds recent reports in memory, for the length of the rendering window (e.g. 15 seconds)
Need to maintain a sliding window of data; can't just store 15 seconds then flush and start from scratch.
When a query process needs to render "live", it fetches the data held by all collectors.
Query process needs to be able to find all collectors (i.e. some sort of service discovery).
We can still save historical data, but at a greatly reduced rate:
Collectors save a report to store once per window period (e.g. once per 15 seconds).
Scope has an optional multitenant deployment model:
Currently, the collectors save all reports to store in near-real-time, and queriers do all rendering by querying the store as-at the time required. Live rendering is done by repeatedly querying as-at "now".
New proposal, which makes the store optional for live rendering:
We can still save historical data, but at a greatly reduced rate:
Fixes #2297