[OBSERVABILITY] Build a metrics component to provide observability into PATH operations

adshmh commented 3 weeks ago

Objective

Create a metrics component to provide visibility into PATH operations. This should provide, at the minimum, visibility into:

Requests from all users
Requests from a specific user
Requests related to a particular service
Requests linked to a particular endpoint

Origin Document

This is required to make it feasible for a gateway operator to get insights into general pattern(s) of the requests being served, as well data specific to one or more services/blockchains.

It is also required for:

Debugging any development issues
Bugs related to request limiting
Bugs related to a particular service

It should act as the foundation (future iterations) of:

Client billing
Mature data pipelines

Goals

Provide the MVP of a metrics component
Establish a single source of truth for both metrics and (future) data pipeline(s); e.g. using a messaging or pubsub platform.
Build the foundation of a metrics component that is straightforward to enhance as the demands from gateway operators evolves.

Deliverables

Selection of new infrastructure
- [ ] Metrics: Selection of a metrics importer/exporter/visualizer; e.g. Grafana/Promethus/Loki/VictoriaMetrics
- [ ] Pubsub: Selection of a messaging platform as single source of truth for metrics and data pipeline(s): e.g. NATS, REDIS, etc.
Infra Changes
- [ ] Support running all infra components locally for development purposes; e.g. Tilt
- [ ] Implementation changes to integrate PATH w/ the messaging platform
Core Logic Changes
- [ ] Implementation to build an operational metrics component.
- [ ] Code changes to enable the exporting of data by PATH: e.g. using Protobuf.
MVP Metrics
- [ ] Identify, define & document the minimum viable set of metrics we need
- [ ] Implement ☝️
Documentation
- [ ] Architecture docs explaining the new infrastructure
- [ ] Quickstart guide on using/exploring the new infrastructure
- [ ] Tutorial on adding new metrics components

Non-goals / Non-deliverables

Exporting and sharing of QoS data: this is a quick follow up.
Data pipeline

Creator: [@adshmh ]

adshmh commented 3 weeks ago

A diagram to show the involved components and their relationship (this will copied to be part of the documentation once the reviews are complete):


flowchart TB
    PubSub[(Messaging Platform / PubSub)]:::PubSub
    U((User)):::user
    GO((Gateway Operator)):::Admin
    DP((Data Pipeline)):::Messaging

    subgraph PI1["PATH Instance"]
        MR[Metrics Reporter]:::Messaging
        COO[Coordinator]:::Messaging
        PME[Prometheus Metrics Endpoint]:::Path
        QI3[QoS Instance]:::Path
        PI3[Protocol Instance]:::Path

        PI3 -- Relay Observations --> COO
        QI3 -- QoS Observations --> COO
        MR -- Metrics --> PME
    end

    COO -. Publishes 1 item per service request .-> PubSub
    PubSub -. Published service requests .-> MR
    PubSub -. Published Service Requests .-> DP
    PME ----> | Metrics | GO
    U ----> | Service Request | COO

    classDef PubSub fill:#f9f,stroke:#333,stroke-width:2px,color:#333;
    classDef Messaging fill:#b2f9fc,stroke:#333,stroke-width:2px,color:#333;
    classDef Path fill:#cfc,stroke:#333,stroke-width:2px,color:#333;
    classDef Admin fill:#ffcccb,stroke:#333,stroke-width:2px,color:#333;
    classDef user fill:#aaaaaf,stroke:#333,stroke-width:2px,color:#333;

adshmh commented 3 weeks ago

cc: @Olshansk @fredteumer @commoddity for review/discussion.

Olshansk commented 3 weeks ago

@adshmh I have not reviewed your diagram yet but I have updated the original PR description.

Overall, it is a good first step but:

Very opinionated based on what we had
Lacks depth on the actual metrics
Missing the importance of local development

Pubsub: Selection of a messaging platform as single source of truth for metrics and data pipeline(s): e.g. NATS, REDIS, etc.

I'm ASSUMING this is to enable down-the-line ETL pipelines with GCP. That's fine but MUST be called out explicitly. Otherwise, I see it as a premature optimization.
I believe THIS ticket should incorporate the scope of work to add a local grafana/prometheus/etc using Tilt. I believe this is something YOU should take on as part of this ticket. Non-trivial amount of work but we can reference & copy-paste poktroll. It'll be a good opportunity to get acquainted with local infra.
While working on this, I want us to define Identify, define & document the minimum viable set of metrics we need together so we know what an end looks like. In particular, While working on (2), I'll help identify a target dashboard. When it's available on LocalNet, we'll mark this as complete.

My goal with these increased scope and requirements is to have a product w/ value at the end rather than just another infrastructural component.

Lmk if you have thoughts, questions, concerns, etc...

I realize this is a lot of work, but we have a good reference, and we will allocate time to it.

buildwithgrove / path