Aaronontheweb
commented
2 years ago These are the definitions for all of our metrics
internal sealed class BuiltInMetrics
{
private BuiltInMetrics(Meter meter)
{
MsgsReceived = meter.CreateCounter<long>(MessagesReceived, MessagesUnit,
"Message-processing throughput by actor type and message type.");
MailboxDepth = meter.CreateHistogram<long>(MetricNames.MailboxDepth, MessagesUnit,
"Number of unprocessed messages waiting inside actor's mailbox.");
DeadLettersReceived = meter.CreateCounter<long>(DeadLetters, MessagesUnit,
"Undeliverable messages sent to /deadletters by message type.");
UnhandledMessagesReceived = meter.CreateCounter<long>(UnhandledMessages,
MessagesUnit,
"Messages that were received by a live actor, but actor was not programmed to handle them, by actor / message type.");
LogMessages = meter.CreateCounter<long>(MetricNames.LogMessages, LogUnits,
"Akka.NET logging events by loglevel.");
ActorRestarts = meter.CreateCounter<long>(MetricNames.ActorRestarts, RestartUnit,
"Actor restarts by type.");
ActorsCreated = meter.CreateCounter<long>(MetricNames.ActorsCreated, CreatedUnit,
"Actors started by type.");
ActorsStopped = meter.CreateCounter<long>(MetricNames.ActorsStopped, StoppedUnit,
"Actors stopped by type.");
MessageReceiveLatency = meter.CreateHistogram<long>(MessageLatency, LatencyUnit,
"Akka.NET message processing latency by actor / message type.");
}There have been some major changes to the Prometheus exporters recently, so I'm wondering if this is an upstream breaking change in the OpenTelemetry project. We've run into some of those lately: https://github.com/open-telemetry/opentelemetry-dotnet/issues/3591
I'll perform some upgrades in our projects and see if I can get to the bottom of it.
ismaelhamed
Arkatufus
Akka: 1.4.40 Phobos: 2.0.6 OpenTelemetry: 1.4.0-alpha.2 Prometheus: 2.37.1 OpenTelemetry-Collector: 0.60.0
@Aaronontheweb I finally figured out why the metrics that we receive from Phobos in Prometheus are different than the ones expected by the Grafana dashboards (hence the dashboards are always empty). When compared with the
Petabridge.Phobos.Webapplication, it looks like we are missing theunit:Our system akka_actor_created akka_actor_mailbox_depth_bucket akka_actor_mailbox_depth_count akka_actor_mailbox_depth_sum akka_cluster_members akka_cluster_reachable akka_logs ...
Petabridge.Phobos.Web akka_actor_created_starts akka_actor_mailbox_depth_msgs_bucket akka_actor_mailbox_depth_msgs_count akka_actor_mailbox_depth_msgs_sum akka_cluster_members_members akka_cluster_reachable_members akka_logs_events ...
Are you guys doing some transformations we are not aware of?
Also notice that we are deploying on bare metal (no k8s).