Scrape requests fail with an exception on Erlang 23

[mkuratczyk]

When using the latest 3.8.4 build with OTP23, RabbitMQ starts up correctly but every request to /metrics results in the following error in the log:

2020-05-15 15:58:13.203 [error] <0.979.0> CRASH REPORT Process <0.979.0> with 0 neighbours crashed with reason: {badarg,[]}
2020-05-15 15:58:13.203 [error] <0.978.0> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.978.0>, stream 1 had its request process <0.979.0> exit with reason badarg and stacktrace []

For those with access to this image, you can reproduce the issue the following way:

$ docker run -p 127.0.0.1:15692:15692/tcp eu.gcr.io/cf-rabbitmq-core/rabbitmq-tanzu@sha256:968fbd203b1b06f537ac5ffddca8a39601115126a560f135510b1416622b49b9
$ curl localhost:15692/metrics # from another terminal of course

[gerhard]

rabbitmq-server-generic-unix-latest-toolchain-3.8.4-alpha.26.tar.xz dev release does not have prometheus.erl v4.6.0, which has this important fix by @dcorbacho for OTP 23: https://github.com/deadtrickster/prometheus.erl/pull/102

As soon as the dev build which is currently flowing through the pipeline gets published, we will release v3.8.4-beta.1 which will fix this issue. Leaving this open until that happens.

cc @michaelklishin

[gerhard]

As soon as https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.8.x/jobs/publish-all-alpha-packages-to-bintray/builds/213 completes, will trigger a beta build (will be later on today).

[michaelklishin]

OK, thanks for clarifying. So there was a relevant memory allocator metric format change in Erlang 23 and the Prometheus client library we use had to be adapted. Since we have bumped the dependency already I suggest that we close it and do some manual testing with a long-running environment before 3.8.4-rc.1 is cut.

[gerhard]

Fixed:

$ curl -s http://127.0.0.1:15692/metrics | grep rabbitmq_build_info
# TYPE rabbitmq_build_info untyped
# HELP rabbitmq_build_info RabbitMQ & Erlang/OTP version info
rabbitmq_build_info{rabbitmq_version="3.8.4-alpha.28",prometheus_plugin_version="3.8.4-alpha.28",prometheus_client_version="4.6.0",erlang_version="23.0"} 1