search

bisq-network / projects

@bisq-network project management
https://bisq.wiki/Project_management
9 stars 2 forks source link

Establish network usage metrics #13

Closed cbeams closed 4 years ago

cbeams commented 4 years ago

Description

See bisq-network/bisq#3916

Citing essential data points and KPIs for desktop client from issue above

Data points to collect within a give timeframe per app version:

  • Number of active nodes (nodes that are part of the network)
  • Number of available offers
  • Number of trades

KPIs needed:

  • Ratio between Number of available offers and Number of active nodes
  • Ratio between Number of trades and Number of available offers

Rationale

If we don't have a baseline and metrics in place to measure our success we'll never be able to tell if some change was a success or a big failure.

As we don't want to include general tracking within the client, which would hurt user privacy, we'll monitor public available information and try to generate as-good-as-it-gets metrics out of it.

Our most important use case is to enable users to buy/sell BTC. Following KPIs help us to measure our success:

Criteria for delivery

KPIs

Within https://monitor.bisq.network/ it should be possible to select a time frame and app version (or all app versions) and display the KPIs above.

Metrics

Within https://monitor.bisq.network/ it should be possible to select a time frame to show the version spread between active nodes

Tasks

here is the tasks list

Estimates

Based on the numbers posted in https://github.com/bisq-network/bisq/issues/3916 this project will need roughly USD 2200 to deliver this project.

ripcurlx commented 4 years ago

@freimair Could you please provide the tasks that are needed to complete this project? Please also the already completed ones. Thanks!

cbeams commented 4 years ago

I'll just add here that the purpose of this project as originally conceived is to support our goal of improving onboarding. We need these (certain, specific) metrics in place, with some historical baseline being established, in order to effectively measure whether our new onboarding workflow implementation is actually helping anything. @ripcurlx, I'd suggest transcribing that and whatever else you see fit to the Rationale section of the description.

ripcurlx commented 4 years ago

@ripcurlx, I'd suggest transcribing that and whatever else you see fit to the Rationale section of the description.

Transcribed the relevant parts from the issue into the Rationale section

freimair commented 4 years ago

here is the tasks list

ripcurlx commented 4 years ago

here is the tasks list

  • [x] design metrics via
  • [x] proof-of-concept via and via
  • [x] create log scraper scripts and monitoring infrastructure
  • [x] unplanned: create one-cmd-installer for general monitoring and scraper script deployment via
  • [ ] deploy data gathering infrastructure
  • [ ] unplanned: fix monitoring daemon (this broke the monitor))
  • [ ] adapt monitoring daemon to collect trade rate data
  • [ ] deploy new monitoring daemon
  • [ ] configure graphs at monitor.bisq.network: KPIs and version spread metrics

Thanks! Regarding the two unplanned task: Does this change the estimated budget above? If yes, by how much?

freimair commented 4 years ago

Thanks! Regarding the two unplanned task: Does this change the estimated budget above? If yes, by how much?

yes, these tasks require an additional 550 USD for me (and 200 for wiz i believe) already acked by @wiz as part of the opsdev budget.

ripcurlx commented 4 years ago

yes, these tasks require an additional 550 USD for me (and 200 for wiz i believe) already acked by @wiz as part of the opsdev budget.

So the total amount for this project is 2750 USD, correct?

freimair commented 4 years ago
task estimate receiver
create the data gathering infrastructure tools USD 500,00 @freimair
update monitoring daemon to deliver additional metrics USD 450,00 @freimair
create installer for general monitoring USD 550,00 @freimair
create pricenode installer (?) USD 200,00 @wiz
deploy the tools 5 * 150 = USD 750,00 price node operator
configure https://monitor.bisq.network USD 500,00 @freimair

which amount belongs to which budget is for you guys to decide.

freimair commented 4 years ago

here is a preliminary set of graphs. Please note that the network size is only provided by one pricenode and is multiplied x8 for a rough estimation of real numbers.

We have to discuss the second KPI, ie. Number of available offers > Number of trades. The number of available offers is a snapshot of a continuous value, the number of trades needs to be accumulated over a certain time frame. What time frame shall it be?

freimair commented 4 years ago

Report Review Meeting 2020-03-19

cbeams commented 4 years ago

It's great to walk through the dashboard and see this data!

cbeams commented 4 years ago

@ripcurlx, can you provide an update on this project?

freimair commented 4 years ago

All the distributed deployments have some spurious issues every now and then. Need to get myself access to a live pricenode to figure out what is going on.

And I need some more budget. Depending on the issue, +550$ should do.

freimair commented 4 years ago

found the error, at least it seems like it. There are no more gaps in the last 30+ days (except for the time where pricenode have been upgraded and the network size data feeds have not been enabled).

@mrosseel's new pricenode does not supply data still. so add a +75 or so to the final number of nodes for numbers after 2020-08-21.

Screenshot from 2020-09-12 11-30-37

ripcurlx commented 4 years ago

Bildschirmfoto 2020-09-30 um 17 33 05

KPIs seem to be stable now. I'll use them for the next releases to check if we made progress. Until now because of missing data every now and then I wasn't able to use them for feature/version evaluation.

I did use the version update metric a lot which is great to see how quickly our users do update.

So from my side this project is now complete.

freimair commented 3 years ago

here is the grafana config

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 13,
  "links": [],
  "panels": [
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 0
      },
      "id": 19,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "alias(scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08), 'active nodes')"
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "estimated Network size (avg of selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "alert": {
        "conditions": [
          {
            "evaluator": {
              "params": [
                100
              ],
              "type": "lt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "name": "Estimated network size",
        "noDataState": "no_data",
        "notifications": []
      },
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 19,
        "x": 5,
        "y": 0
      },
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": 0,
          "refId": "A",
          "target": "alias(scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08), 'active nodes')"
        },
        {
          "hide": true,
          "refCount": 0,
          "refId": "B",
          "target": "aliasSub(scale(movingWindow(groupByNode(servers.*.requestsPer750Seconds.*, 0, 'sum'), '15min', 'last'), 0.08), 'servers.([a-z0-9]+)\\..*gauge-(.*?),', '\\2-\\1')",
          "textEditor": false
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "C",
          "target": "aliasByNode(scale(groupByNode(movingWindow(removeEmptySeries(servers.*.requestsPer750Seconds.*), '15min', 'average'), 1, 'sum'), 0.08), 0)",
          "textEditor": false
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "fill": true,
          "line": true,
          "op": "lt",
          "value": 100,
          "yaxis": "left"
        }
      ],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "estimated Network size",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 3
      },
      "id": 18,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "A",
          "target": "scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08)"
        },
        {
          "refCount": 0,
          "refId": "C",
          "target": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), #A)",
          "targetFull": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08))"
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "offers per node (avg of selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 1,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 6
      },
      "id": 17,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "C",
          "target": "keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200)"
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "B",
          "target": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), #C)",
          "targetFull": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200))",
          "textEditor": false
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "trades per offer (avg over selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 9
      },
      "id": 4,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "A",
          "target": "scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08)",
          "textEditor": false
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "C",
          "target": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), #A)",
          "targetFull": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08))"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "offers per node",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 9
      },
      "id": 5,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "C",
          "target": "keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200)"
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "B",
          "target": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), #C)",
          "targetFull": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200))",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "trades per offer (6h avg)",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 18
      },
      "id": 16,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "refCount": 0,
          "refId": "A",
          "target": "aliasByMetric(sumSeries(removeEmptySeries(bisq.P2PMarketStats.*.*.*)))"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Open Offers",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "cacheTimeout": null,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 18
      },
      "id": 13,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6hours')",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Number of Trades (6h avg)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "cacheTimeout": "",
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 27
      },
      "id": 3,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "sideWidth": 65,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": true,
      "pluginVersion": "6.2.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "aliasSub(aliasSub(exclude(aliasByNode(sortBy(asPercent(movingAverage(groupByNode(servers.*.requestsPer750Seconds.*, 3, 'sum'), '15min')), 'last', false), 0), 'vget'), '^.*(v\\d.\\d.\\d).*', '\\1'), '_', '.')",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "version spread",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 27
      },
      "id": 14,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": true,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "aliasSub(aliasByNode(sortBy(asPercent(movingAverage(groupByNode(servers.*.hsversionStats.*, 3, 'sum'), '15min')), 'last', false), 0), '^.*(HSv\\d)', '\\1')",
          "textEditor": false
        },
        {
          "hide": true,
          "refCount": 0,
          "refId": "B",
          "target": "groupByNode(servers.*.hsversionStats.*, 3, 'average')"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "hs version spread",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": false,
  "schemaVersion": 18,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-24h",
    "to": "now"
  },
  "timepicker": {
    "nowDelay": "",
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "Key Performance Indicators",
  "uid": "dEVJDduWz",
  "version": 62
}