search

kamon-io / kamon-akka-remote

Kamon Instrumentation for Akka Remote
https://kamon.io/docs/latest/instrumentation/akka/
Other
5 stars 10 forks source link

Confusing akka-sharding metrics values #22

Open sebarys opened 5 years ago

sebarys commented 5 years ago

Hi, I've just upgraded kamon-akka-remote dependency to play with metrics about sharding. After deploying changes to test cluster I've tried to check returned values and compare them with e.g. information about sharding exposed in akka management extension API

I have cluster with 3 nodes, below informations about shard region with Distpacher name on each node: 

curl 10.244.16.84:8558/cluster/shards/Dispatcher
{"regions":[{"numEntities":0,"shardId":"34"},{"numEntities":0,"shardId":"12"},{"numEntities":0,"shardId":"23"},{"numEntities":0,"shardId":"40"},{"numEntities":0,"shardId":"11"},{"numEntities":1,"shardId":"44"},{"numEntities":1,"shardId":"33"},{"numEntities":0,"shardId":"56"},{"numEntities":0,"shardId":"37"},{"numEntities":0,"shardId":"61"},{"numEntities":0,"shardId":"13"},{"numEntities":1,"shardId":"24"},{"numEntities":0,"shardId":"16"},{"numEntities":0,"shardId":"59"},{"numEntities":0,"shardId":"21"},{"numEntities":0,"shardId":"32"},{"numEntities":0,"shardId":"39"},{"numEntities":0,"shardId":"14"},{"numEntities":1,"shardId":"58"},{"numEntities":0,"shardId":"20"},{"numEntities":1,"shardId":"27"},{"numEntities":0,"shardId":"2"},{"numEntities":1,"shardId":"18"}]}

curl 10.244.30.185:8558/cluster/shards/Dispatcher
{"regions":[{"numEntities":0,"shardId":"98"},{"numEntities":0,"shardId":"93"},{"numEntities":0,"shardId":"66"},{"numEntities":0,"shardId":"84"},{"numEntities":1,"shardId":"8"},{"numEntities":0,"shardId":"62"},{"numEntities":0,"shardId":"90"},{"numEntities":0,"shardId":"68"},{"numEntities":0,"shardId":"83"},{"numEntities":0,"shardId":"79"},{"numEntities":1,"shardId":"72"},{"numEntities":1,"shardId":"76"},{"numEntities":0,"shardId":"65"},{"numEntities":0,"shardId":"80"},{"numEntities":0,"shardId":"82"},{"numEntities":0,"shardId":"96"},{"numEntities":0,"shardId":"64"},{"numEntities":0,"shardId":"75"},{"numEntities":0,"shardId":"70"},{"numEntities":1,"shardId":"81"},{"numEntities":0,"shardId":"7"},{"numEntities":0,"shardId":"97"},{"numEntities":0,"shardId":"74"},{"numEntities":0,"shardId":"85"}]}

curl 10.244.49.18:8558/cluster/shards/Dispatcher
{"regions":[{"numEntities":0,"shardId":"45"},{"numEntities":1,"shardId":"51"},{"numEntities":0,"shardId":"78"},{"numEntities":0,"shardId":"4"},{"numEntities":0,"shardId":"77"},{"numEntities":1,"shardId":"46"},{"numEntities":0,"shardId":"99"},{"numEntities":1,"shardId":"35"},{"numEntities":0,"shardId":"5"},{"numEntities":1,"shardId":"87"},{"numEntities":0,"shardId":"57"},{"numEntities":0,"shardId":"6"},{"numEntities":0,"shardId":"36"},{"numEntities":0,"shardId":"60"},{"numEntities":0,"shardId":"69"},{"numEntities":0,"shardId":"95"},{"numEntities":0,"shardId":"53"},{"numEntities":0,"shardId":"42"},{"numEntities":1,"shardId":"86"},{"numEntities":1,"shardId":"29"},{"numEntities":0,"shardId":"41"},{"numEntities":1,"shardId":"3"},{"numEntities":0,"shardId":"91"},{"numEntities":0,"shardId":"52"}]}

Informations from kamon-akka-remote sharding metrics returned using Prometheus integration:

curl 10.244.16.84:9090/metrics | grep sharding | grep Dispatcher
akka_cluster_sharding_region_processed_messages_total{type="Dispatcher"} 31903.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="10.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="30.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="100.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="300.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="1000.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="3000.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="10000.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="30000.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="100000.0"} 614882.0
akka_cluster_sharding_shard_hosted_entities_bucket{type="Dispatcher",le="+Inf"} 614882.0
akka_cluster_sharding_shard_hosted_entities_count{type="Dispatcher"} 614882.0
akka_cluster_sharding_shard_hosted_entities_sum{type="Dispatcher"} 126644.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="10.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="30.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="100.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="300.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="1000.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="3000.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="10000.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="30000.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="100000.0"} 26780.0
akka_cluster_sharding_region_hosted_entities_bucket{type="Dispatcher",le="+Inf"} 26780.0
akka_cluster_sharding_region_hosted_entities_count{type="Dispatcher"} 26780.0
akka_cluster_sharding_region_hosted_entities_sum{type="Dispatcher"} 126644.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="10.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="30.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="100.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="300.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="1000.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="3000.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="10000.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="30000.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="100000.0"} 614882.0
akka_cluster_sharding_shard_processed_messages_bucket{type="Dispatcher",le="+Inf"} 614882.0
akka_cluster_sharding_shard_processed_messages_count{type="Dispatcher"} 614882.0
akka_cluster_sharding_shard_processed_messages_sum{type="Dispatcher"} 0.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="10.0"} 1501.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="30.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="100.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="300.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="1000.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="3000.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="10000.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="30000.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="100000.0"} 1007502.0
akka_cluster_sharding_region_hosted_shards_bucket{type="Dispatcher",le="+Inf"} 1007502.0
akka_cluster_sharding_region_hosted_shards_count{type="Dispatcher"} 1007502.0
akka_cluster_sharding_region_hosted_shards_sum{type="Dispatcher"} 23134531.0

I'm not able to analyse this metrics and get any meaningful informations from them. Is it possible to receive some guidance what this numbers stand for?

Can I somehow extract how many entities are in each node based on metrics from kamon-akka-remote? From akka mgmt API I see that on first node I have 7 shard entities:

Thank you for any insights or advice

Regards, Sebastian

mbuccini commented 4 years ago

Hi,

the way you have your metrics, you can't differentiate them by node. You will need to add a tag to each metrics for each node you get them from. Finally, you'll need to have something like this:

akka_cluster_sharding_region_hosted_shards_count{type="Dispatcher", my_node_tag="node-1"} 1007502.0
akka_cluster_sharding_region_hosted_shards_count{type="Dispatcher",  my_node_tag="node-2"} 95502.0
akka_cluster_sharding_region_hosted_shards_count{type="Dispatcher", my_node_tag="node-3"} 567502.0

Finally you will be able to aggregate:

rate(akka_cluster_sharding_region_hosted_shards_sum{type="Dispatcher", my_node_tag="node-1"}[interval]) / rate(akka_cluster_sharding_region_hosted_shards_count{type="Dispatcher", my_node_tag="node-1"}[interval])