Add wallet_key_count metric

[qustavo]

This PR adds a new wallet metric: _wallet_keycount which reports the current number of keys derived per account. This metric is particularly useful when running a recovery with large gaps.

When recovering a node (see recovery.md) we need to specify a recovery window value:

The recovery window is a metric that the on-chain rescanner will use to determine when all the "used" addresses have been found. If the recovery window is two, lnd will fail to find funds in any addresses generated after the point in which two consecutive addresses were generated but never used. If an lnd on-chain wallet was extensively used, then users may want to increase the default value.

To find an accurate recovery window we could rely on the wallet key count (see https://github.com/lightningnetwork/lnd/issues/5767).

This PR also bumps lndclient to :lnd-13-0 (https://github.com/lightninglabs/lndclient/tree/lnd-13-0) which introduces the ListAccounts call.

[qustavo]

Any interest in adding these metrics to our provisioned grafana dashboards so they're more easily available?

how? is that dashboard open/public?

[guggero]

Yes, they're here: https://github.com/lightninglabs/lndmon/tree/master/grafana/provisioning/dashboards

[qustavo]

Yes, they're here: https://github.com/lightninglabs/lndmon/tree/master/grafana/provisioning/dashboards

Thanks, I added the metric to a new panel

[carlaKC]

I would suggest something like:

{
  "type": "graph",
  "title": "Address Types",
  "gridPos": {
    "x": 12,
    "y": 16,
    "w": 12,
    "h": 9
  },
  "datasource": "Prometheus",
  "aliasColors": {},
  "bars": true,
  "dashLength": 10,
  "dashes": false,
  "description": "",
  "fill": 1,
  "fillGradient": 0,
  "hiddenSeries": false,
  "id": 13,
  "legend": {
    "avg": false,
    "current": false,
    "max": false,
    "min": false,
    "show": false,
    "total": false,
    "values": false
  },
  "lines": false,
  "linewidth": 1,
  "nullPointMode": "null",
  "options": {
    "dataLinks": []
  },
  "percentage": false,
  "pointradius": 2,
  "points": false,
  "renderer": "flot",
  "seriesOverrides": [],
  "spaceLength": 10,
  "stack": false,
  "steppedLine": false,
  "targets": [
    {
      "refId": "A",
      "expr": "lnd_wallet_key_count",
      "intervalFactor": 1,
      "format": "time_series",
      "legendFormat": "{{address_type}}"
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeRegions": [],
  "timeShift": null,
  "tooltip": {
    "shared": false,
    "sort": 0,
    "value_type": "individual"
  },
  "xaxis": {
    "buckets": null,
    "mode": "series",
    "name": null,
    "show": true,
    "values": [
      "max"
    ]
  },
  "yaxes": [
    {
      "format": "short",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": true
    },
    {
      "format": "short",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": true
    }
  ],
  "yaxis": {
    "align": false,
    "alignLevel": null
  },
  "links": []
}

{
  "type": "graph",
  "title": "Address Types",
  "gridPos": {
    "x": 0,
    "y": 25,
    "w": 12,
    "h": 9
  },
  "datasource": "Prometheus",
  "aliasColors": {},
  "bars": true,
  "dashLength": 10,
  "dashes": false,
  "description": "",
  "fill": 1,
  "fillGradient": 0,
  "hiddenSeries": false,
  "id": 12,
  "legend": {
    "avg": false,
    "current": false,
    "max": false,
    "min": false,
    "show": false,
    "total": false,
    "values": false
  },
  "lines": false,
  "linewidth": 1,
  "nullPointMode": "null",
  "options": {
    "dataLinks": []
  },
  "percentage": false,
  "pointradius": 2,
  "points": false,
  "renderer": "flot",
  "seriesOverrides": [],
  "spaceLength": 10,
  "stack": false,
  "steppedLine": false,
  "targets": [
    {
      "refId": "A",
      "expr": "sum(lnd_wallet_key_count{key_type=\"internal\"})",
      "intervalFactor": 1,
      "format": "time_series",
      "legendFormat": "Internal Key Count"
    },
    {
      "refId": "B",
      "expr": "sum(lnd_wallet_key_count{key_type=\"external\"})",
      "intervalFactor": 1,
      "format": "time_series",
      "legendFormat": "External Key Count"
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeRegions": [],
  "timeShift": null,
  "tooltip": {
    "shared": false,
    "sort": 0,
    "value_type": "individual"
  },
  "xaxis": {
    "buckets": null,
    "mode": "series",
    "name": null,
    "show": true,
    "values": [
      "max"
    ]
  },
  "yaxes": [
    {
      "format": "short",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": true
    },
    {
      "format": "short",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": true
    }
  ],
  "yaxis": {
    "align": false,
    "alignLevel": null
  },
  "links": []
}

You'd need to update this json a bit (datasource, namespace, better titles etc), but should get you most of the way there.

[qustavo]

You are suggesting two different panels? or both graphs belong to the same panel? Also, I'm using "datasource": "$datasource" following what it's already in the rest of the dashboards, in your example it is using "datasource": "Prometheus", is that changed on purpose?

[carlaKC]

Sorry somehow completely missed this comment!

You are suggesting two different panels?

Was thinking 2x panels?

"datasource": "Prometheus", is that changed on purpose?

Nope, should be "$datasource", those are just my local deafults.

[qustavo]

@carlaKC I just noticed that and it happened when I started using lnd-0.14.0. Earlier versions only have a default account but 0.14.0 now has more accounts (named act:<N>) . So my initial chart is rendering well, I'll need to investigate and see what's the best way to display this info.

[qustavo]

After filtering out the non-default wallets in the chart this is the result:

Also, are they all the collected metrics rendered as charts? do we need to do that for this one?

[carlaKC]

@carlaKC I just noticed that and it happened when I started using lnd-0.14.0. Earlier versions only have a default account but 0.14.0 now has more accounts (named act:<N>) . So my initial chart is rendering well, I'll need to investigate and see what's the best way to display this info.

Ah interesting! Difficult to display succinctly.

We could just change the existing chart to a graph and go with something like this: It's not pretty, but has all the information exposed at least.

Also, are they all the collected metrics rendered as charts? do we need to do that for this one?

Generally metrics are, but it's not a strict requirement. If you want to just drop the grafana change, we can just merge the addition of the metric to prometheus!

[qustavo]

I prefer to drop the changes to Grafana since it is not pretty to display, and different users might need different chart details/accounts. Will update the PR now

[lightninglabs-deploy]

@ellemouton: review reminder

[ellemouton]

thanks for the quick update @qustavo 🚀 almost there, just need to also rebase master as your pr now includes some other commits