Dynamically generate the diagram

[rehandalal]

It might be nice to stop using a fixed SVG and instead dynamically generate the diagram using something like JointJS

[leplatrem]

What would be the input to generate the diagram?

[mostlygeek]

Had a conversation with @rehandalal about this:

Context: We are building out more services that report telemetry about how things are working on the client's machine. The diagram in Poucave's UI is a great and easy way to quickly see how the system is behaving. However, the current poucave UI is a "team view" for the product delivery team and the systems they were interested in.

Poucave supporting more "team views" would make it more useful to monitor other systems like sync, push, etc.

[leplatrem]

There are different ways to achieve this.

Adding views to a single instance, or run multiple instances.

Since running Poucave is pretty cheap, I would have a preference for the latter :shrug: And if we want to integrate them all together, then one instance can use the other instances endpoints to report their status :exploding_head:

[mostlygeek]

I like the multiple instances approach.

Poucave is cheap to run but expensive to set up. The setup is expensive because it's not automated and requires getting ops to configure each one. If we can make the setup/deployment process as easy as adding/updating a configuration we could keep poucave simple (+1) and still get custom pages.

I have to admit, the idea of Poucave checks building on other poucave checks frightens me a bit. :)

[leplatrem]

If we can make the setup/deployment process as easy as adding/updating a configuration we could keep poucave simple (+1) and still get custom pages.

I agree. Let's see, concretely to setup a new instance, currently we need to:

• implement (if any missing) the specific checks
• write the .toml configuration file to leverage the desired checks
• draw the SVG and annotate it with the specific object id syntax
• deploy the app with the 2 config.toml & diagram.svg files

This ticket originally suggested that we could generate the diagram from the config file (indicating links and labels for example).

Now, maybe that the idea of having different views could be closer to issue #373 ?

[mostlygeek]

Regarding tags, tags were originally designed to create a single "up or down?" endpoint for a group of checks. They weren't designed for what this issue is about, or even creating custom UI for team views. @rehandalal changed my thinking here.

---

Thinking about it more, I would argue that poucave now has two purposes:

1. It's original purpose: platform for creating complex checks for detecting systemic failures.
2. A status page to understand what's going on in the system as a whole.

For the second purpose, the SVG is so valuable because it gives context to the checks. It helps people understand the system quickly:

I think this is really the problem we're solving: People want to their own custom poucave page (svg, checks, etc) but it's not easy enough to get a custom one.

How can we make poucave more self serve so people can:

• self serve
• land new checks
• create and update their page with an automated flow
• have a place to ask for help (slack, #delivery?) when they're stuck

[leplatrem]

How many and how often do you think we'd have to create new pages?

Things to consider:

• drawing a SVG is a piece of cake
• creating a config.toml with the list of checks is pretty straightforward (see example)
• landing new checks is already self-serve (pull-requests in this repo). Plus the checks can come from other repos as long as they are available in the PYTHONPATH.
• A single poucave instance cannot scale indefinitely, because it uses in-memory cache and thus a single HTTP worker

Some possible approaches:

• have an easy to use script/template to create new poucave instances in cloudops-infra (svg + config + domainname as input)
• change the deployment recipe in cloudops-infra so that it would deploy one [instance|url|domain] for each folder containing the SVG & config files (instead of a single one)
• bloat the current code to add notions of views and pages

[mostlygeek]

How many and how often do you think we'd have to create new pages?

One or two for 2021 would be my estimate. It's not worth automating now. However, when we have a few more customers and use cases on poucave it will get the attention of other teams. At that point putting effort into making it more self serve will accelerate its adoption.