[🐛 BUG]: Duplicate Metrics Collector Registration Attempted

[L3tum]

No duplicates 🥲.

• [X] I have searched for a similar issue in our bug tracker and didn't find any solutions.

What happened?

Expected: RR starts 😆

Actual: On start we get a multitude of errors logged such as "metric with provided name already exist". As far as I know this seems to be "intended" or at least okay behaviour. However, sometimes, often multiple times, we get the error "Error 'metrics_plugin_declare: duplicate metrics collector registration attempted' on tcp://127.0.0.1:6001" and RR crashes hard.

This seems to have been tried to suppres in roadrunner-metrics but doesn't seem to work?

I do not think this was an issue in earlier versions, but we made quiet the jump recently in versions so I can't quite tell which one wasn't affected (or if we just didn't trigger this problem).

In case it makes a difference, we're using the Baldinof Bundle for the Symfony Integration.

Version (rr --version)

2.10.7

Relevant log output

Spiral\RoadRunner\Metrics\Exception\MetricsException {#116

#message: "Error 'metrics_plugin_declare: duplicate metrics collector registration attempted' on tcp://127.0.0.1:6001"

#code: 0

#file: "./vendor/spiral/roadrunner-metrics/src/Metrics.php"

#line: 102

-previous: Spiral\Goridge\RPC\Exception\ServiceException {#109

#message: "Error 'metrics_plugin_declare: duplicate metrics collector registration attempted' on tcp://127.0.0.1:6001"

#code: 0

#file: "./vendor/spiral/goridge/src/RPC/RPC.php"

#line: 151

trace: {

./vendor/spiral/goridge/src/RPC/RPC.php:151 { …}

./vendor/spiral/goridge/src/RPC/RPC.php:94 { …}

./vendor/spiral/roadrunner-metrics/src/Metrics.php:94 { …}

./vendor/baldinof/roadrunner-bundle/src/EventListener/DeclareMetricsListener.php:79 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:270 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:230 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:59 { …}

./vendor/baldinof/roadrunner-bundle/src/Worker/Worker.php:98 { …}

./vendor/baldinof/roadrunner-bundle/src/Runtime/Runner.php:27 { …}

./vendor/autoload_runtime.php:29 { …}

./public/index.php:7 {

›

› require_once dirname(__DIR__) . '/vendor/autoload_runtime.php';

›

arguments: {

"/var/www/vendor/autoload_runtime.php"

}

}

}

}

trace: {

./vendor/spiral/roadrunner-metrics/src/Metrics.php:102 { …}

./vendor/baldinof/roadrunner-bundle/src/EventListener/DeclareMetricsListener.php:79 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:270 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:230 { …}

./vendor/symfony/event-dispatcher/EventDispatcher.php:59 { …}

./vendor/baldinof/roadrunner-bundle/src/Worker/Worker.php:98 { …}

./vendor/baldinof/roadrunner-bundle/src/Runtime/Runner.php:27 { …}

./vendor/autoload_runtime.php:29 { …}

./public/index.php:7 {

›

› require_once dirname(__DIR__) . '/vendor/autoload_runtime.php';

›

arguments: {

"/var/www/vendor/autoload_runtime.php"

}

}

}

}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "x", "type": "counter", "namespace": ""}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "xxxxxxxx", "type": "counter", "namespace": ""}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "xxxxx", "type": "counter", "namespace": ""}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "xx", "type": "counter", "namespace": ""}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "xxxx", "type": "counter", "namespace": ""}

2022-07-21T11:01:43.648Z    ERROR   metrics         metric with provided name already exist {"name": "xxx", "type": "histogram", "namespace": ""}

2022-07-21T11:01:43.649Z    ERROR   container/poller.go:16  vertex got an error {"id": "http.Plugin", "error": "static_pool_allocate_workers: WorkerAllocate: EOF"}

github.com/roadrunner-server/endure/pkg/container.(*Endure).poll.func1

github.com/roadrunner-server/endure@v1.3.0/pkg/container/poller.go:16

error: static_pool_allocate_workers: WorkerAllocate: EOF

plugin: http.Plugin

[rustatian]

Hey @L3tum 👋🏻 I don't think this is RR related issue because it relates to the roadrunner-worker.

Since RR start several workers, they are trying to declare the same metrics. To sound metrics only once, you may try to use a server.on_init feature (to declare metrics in the prometheus). And then you may use it in your app.

EDIT: https://github.com/roadrunner-server/roadrunner/blob/master/.rr.yaml#L28

[L3tum]

Hey @rustatian!

Thanks for the info. So this is essentially expected behaviour since we, for example, start 8 workers they'll try to declare the same metrics and fail?

It seems to be a bit weird to me that we haven't triggered this behaviour before when we've been using 2-4 workers, and only now with 8 workers it happens somewhat reliably.

It also seems like the Baldinof Bundle uses it without any synchronization and its docs encourage the use of it that way, so if the case is really that we'd need to use a server.on_init script then I'd classify that as a bug in the bundle as well.

[rustatian]

Thanks for the info. So this is essentially expected behaviour since we, for example, start 8 workers they'll try to declare the same metrics and fail?

I'll check; I might skip these errors on the RR's side (if possible).

It also seems like the Baldinof Bundle uses it without any synchronization and its docs encourage the use of it that way, so if the case is really that we'd need to use a server.on_init script then I'd classify that as a bug in the bundle as well.

Yeah, please, try the on_init. We will update the docs to include that note if it would be fine.

[L3tum]

I'll check; I might skip these errors on the RR's side (if possible).

Thanks!

It seems to be possible to declare the metrics without the on_init via the rr-config, right?

[rustatian]

I'll check; I might skip these errors on the RR's side (if possible).

Thanks!

It seems to be possible to declare the metrics without the on_init via the rr-config, right?

Yeah, sure. I guessed that you needed to declare them from the PHP, but if not, feel free to use the config 😃

[L3tum]

Cool, thank you for your help! 😄

[rustatian]

My pleasure. Feel free to close the bug if that wouldn't be an issue with declaring via configuration.

[L3tum]

Seems to be working well and the duplicate metric errors are also gone. Thanks again, I'll close the issue