terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

[terafoundation] Prom metrics exporter doesn't reset metrics between updates #3747

Closed busma13 closed 2 months ago

busma13 commented 2 months ago

This PR makes the following changes:

ref: #3743

godber commented 2 months ago

Uh, I completely forgot about this ...

https://github.com/terascope/teraslice/commit/49d356cf7dc6583c518f98a4500355c53e173e9f

curl -H "Accept: application/openmetrics-text;" -sS http://localhost:5678/cluster/stats
# TYPE teraslice_slices_processed counter
teraslice_slices_processed{cluster="teraslice-dev1"} 2
# TYPE teraslice_slices_failed counter
teraslice_slices_failed{cluster="teraslice-dev1"} 0
# TYPE teraslice_slices_queued counter
teraslice_slices_queued{cluster="teraslice-dev1"} 0
# TYPE teraslice_workers_joined counter
teraslice_workers_joined{cluster="teraslice-dev1"} 1
# TYPE teraslice_workers_disconnected counter
teraslice_workers_disconnected{cluster="teraslice-dev1"} 0
# TYPE teraslice_workers_reconnected counter
teraslice_workers_reconnected{cluster="teraslice-dev1"} 0

compared to:

curl -sS http://localhost:5678/cluster/stats
{
    "controllers": {
        "processed": 2,
        "failed": 0,
        "queued": 0,
        "job_duration": 3,
        "workers_joined": 1,
        "workers_disconnected": 0,
        "workers_reconnected": 0
    },
    "slicer": {
        "processed": 2,
        "failed": 0,
        "queued": 0,
        "job_duration": 3,
        "workers_joined": 1,
        "workers_disconnected": 0,
        "workers_reconnected": 0
    }
}

We should document the existence of this capability and we should really consider expanding the labels coming out of it or consolidating these metrics in with the new main built in exporter.

busma13 commented 2 months ago

teraslice_slices_processed{cluster="teraslice-dev1"} 2

curl localhost:5678
{
    "arch": "arm64",
    "clustering_type": "kubernetes",
    "name": "ts-dev1",
    "node_version": "v18.20.4",
    "platform": "linux",
    "teraslice_version": "v2.3.0"
}

cluster === teraslice.name from the terafoundation, correct? Which is what is returned in the request above.

godber commented 2 months ago

cluster === teraslice.name from the terafoundation, correct? Which is what is returned in the request above.

Correct

busma13 commented 2 months ago

We should document the existence of this capability

I added an example in the endpoints-json.md file