search

sensu-plugins / sensu-plugins-docker

This plugin provides native Docker instrumentation for monitoring and metrics collection, including: container status, container number, and container metrics via `docker ps`.
http://sensu-plugins.io
MIT License
35 stars 57 forks source link

metrics-docker-stats.rb invalid line in graphite #29

Closed celevra closed 8 years ago

celevra commented 8 years ago

Hi,

i get 

invalid message line: stats.docker.meet-3.precpu_stats.cpu_usage.percpu_usage

but i don't see whats wrong:

02/06/2016 13:26:46 :: invalid message line: stats.docker.meet-3.cpu_stats.cpu_usage.percpu_usage 87947260773 89019969698 87542041490 88047874778 19494320623 20385737322 20330167000 21439983103 1464866793

is there anyhting i can provide?

regards

eheydrick commented 8 years ago

Can you provide the full output from the plugin. I wonder if there's an extra newline or something like that? Or it doesn't like the dash.

celevra commented 8 years ago

here comes all relevant log i think:

dev ~ # cat /etc/sensu/conf.d/metrics_docker_core.json
{
  "checks": {
    "metrics_docker_core": {
      "type": "metric",
      "handlers": ["graphite"],
      "command": "/usr/local/bin/metrics-docker-stats.rb -H /var/run/docker.sock -p unix -n -s stats.docker",
      "interval": 60,
      "subscribers": [ "ALL" ]
    }
  }
}
dev ~ # /usr/local/bin/metrics-docker-stats.rb -H /var/run/docker.sock -p unix -n -s stats.docker
stats.docker.registry.precpu_stats.cpu_usage.total_usage 0 1465019117
stats.docker.registry.precpu_stats.cpu_usage.percpu_usage
stats.docker.registry.precpu_stats.cpu_usage.usage_in_kernelmode 0 1465019117
stats.docker.registry.precpu_stats.cpu_usage.usage_in_usermode 0 1465019117
stats.docker.registry.precpu_stats.system_cpu_usage 0 1465019117
stats.docker.registry.precpu_stats.throttling_data.periods 0 1465019117
stats.docker.registry.precpu_stats.throttling_data.throttled_periods 0 1465019117
stats.docker.registry.precpu_stats.throttling_data.throttled_time 0 1465019117
stats.docker.registry.cpu_stats.cpu_usage.total_usage 10595120246 1465019117
stats.docker.registry.cpu_stats.cpu_usage.percpu_usage 5381824159 2050945234 2470029525 692321328 1465019117
stats.docker.registry.cpu_stats.cpu_usage.usage_in_kernelmode 420000000 1465019117
stats.docker.registry.cpu_stats.cpu_usage.usage_in_usermode 3840000000 1465019117
stats.docker.registry.cpu_stats.system_cpu_usage 1266037310000000 1465019117
stats.docker.registry.cpu_stats.throttling_data.periods 0 1465019117
stats.docker.registry.cpu_stats.throttling_data.throttled_periods 0 1465019117
stats.docker.registry.cpu_stats.throttling_data.throttled_time 0 1465019117
stats.docker.registry.memory_stats.usage 14454784 1465019117
stats.docker.registry.memory_stats.max_usage 31076352 1465019117
stats.docker.registry.memory_stats.stats.active_anon 7340032 1465019117
stats.docker.registry.memory_stats.stats.active_file 307200 1465019117
stats.docker.registry.memory_stats.stats.cache 7131136 1465019117
stats.docker.registry.memory_stats.stats.hierarchical_memory_limit 18446744073709551615 1465019117
stats.docker.registry.memory_stats.stats.inactive_anon 0 1465019117
stats.docker.registry.memory_stats.stats.inactive_file 6807552 1465019117
stats.docker.registry.memory_stats.stats.mapped_file 1835008 1465019117
stats.docker.registry.memory_stats.stats.pgfault 7622 1465019117
stats.docker.registry.memory_stats.stats.pgmajfault 104 1465019117
stats.docker.registry.memory_stats.stats.pgpgin 7388 1465019117
stats.docker.registry.memory_stats.stats.pgpgout 4881 1465019117
stats.docker.registry.memory_stats.stats.rss 7323648 1465019117
stats.docker.registry.memory_stats.stats.rss_huge 2097152 1465019117
stats.docker.registry.memory_stats.stats.total_active_anon 7340032 1465019117
stats.docker.registry.memory_stats.stats.total_active_file 307200 1465019117
stats.docker.registry.memory_stats.stats.total_cache 7131136 1465019117
stats.docker.registry.memory_stats.stats.total_inactive_anon 0 1465019117
stats.docker.registry.memory_stats.stats.total_inactive_file 6807552 1465019117
stats.docker.registry.memory_stats.stats.total_mapped_file 1835008 1465019117
stats.docker.registry.memory_stats.stats.total_pgfault 7622 1465019117
stats.docker.registry.memory_stats.stats.total_pgmajfault 104 1465019117
stats.docker.registry.memory_stats.stats.total_pgpgin 7388 1465019117
stats.docker.registry.memory_stats.stats.total_pgpgout 4881 1465019117
stats.docker.registry.memory_stats.stats.total_rss 7323648 1465019117
stats.docker.registry.memory_stats.stats.total_rss_huge 2097152 1465019117
stats.docker.registry.memory_stats.stats.total_unevictable 0 1465019117
stats.docker.registry.memory_stats.stats.total_writeback 0 1465019117
stats.docker.registry.memory_stats.stats.unevictable 0 1465019117
stats.docker.registry.memory_stats.stats.writeback 0 1465019117
stats.docker.registry.memory_stats.failcnt 0 1465019117
stats.docker.registry.memory_stats.limit 12600770560 1465019117
stats.docker.registry.networks.eth0.rx_bytes 110367 1465019117
stats.docker.registry.networks.eth0.rx_packets 870 1465019117
stats.docker.registry.networks.eth0.rx_errors 0 1465019117
stats.docker.registry.networks.eth0.rx_dropped 0 1465019117
stats.docker.registry.networks.eth0.tx_bytes 262883 1465019117
stats.docker.registry.networks.eth0.tx_packets 836 1465019117
stats.docker.registry.networks.eth0.tx_errors 0 1465019117
stats.docker.registry.networks.eth0.tx_dropped 0 1465019117

dev ~ #
dev ~ # tail -f /var/log/carbon/listener.log
04/06/2016 07:43:34 :: Message received: Method(name=deliver, id=60) ('graphite_consumer', 139, False, 'metrics', '') content = <Content instance: body='stats.docker.registry.precpu_stats.cpu_usage.total_usage 0 1465019013\nstats.docker.registry.precpu_stats.cpu_usage.percpu_usage\nstats.docker.registry.precpu_stats.cpu_usage.usage_in_kernelmode 0 1465019013\nstats.docker.registry.precpu_stats.cpu_usage.usage_in_usermode 0 1465019013\nstats.docker.registry.precpu_stats.system_cpu_usage 0 1465019013\nstats.docker.registry.precpu_stats.throttling_data.periods 0 1465019013\nstats.docker.registry.precpu_stats.throttling_data.throttled_periods 0 1465019013\nstats.docker.registry.precpu_stats.throttling_data.throttled_time 0 1465019013\nstats.docker.registry.cpu_stats.cpu_usage.total_usage 10576789225 1465019013\nstats.docker.registry.cpu_stats.cpu_usage.percpu_usage 5380510422 2033927950 2470029525 692321328 1465019013\nstats.docker.registry.cpu_stats.cpu_usage.usage_in_kernelmode 410000000 1465019013\nstats.docker.registry.cpu_stats.cpu_usage.usage_in_usermode 3840000000 1465019013\nstats.docker.registry.cpu_stats.system_cpu_usage 1265626750000000 1465019013\nstats.docker.registry.cpu_stats.throttling_data.periods 0 1465019013\nstats.docker.registry.cpu_stats.throttling_data.throttled_periods 0 1465019013\nstats.docker.registry.cpu_stats.throttling_data.throttled_time 0 1465019013\nstats.docker.registry.memory_stats.usage 14454784 1465019013\nstats.docker.registry.memory_stats.max_usage 31076352 1465019013\nstats.docker.registry.memory_stats.stats.active_anon 7340032 1465019013\nstats.docker.registry.memory_stats.stats.active_file 307200 1465019013\nstats.docker.registry.memory_stats.stats.cache 7131136 1465019013\nstats.docker.registry.memory_stats.stats.hierarchical_memory_limit 18446744073709551615 1465019013\nstats.docker.registry.memory_stats.stats.inactive_anon 0 1465019013\nstats.docker.registry.memory_stats.stats.inactive_file 6807552 1465019013\nstats.docker.registry.memory_stats.stats.mapped_file 1835008 1465019013\nstats.docker.registry.memory_stats.stats.pgfault 7622 1465019013\nstats.docker.registry.memory_stats.stats.pgmajfault 104 1465019013\nstats.docker.registry.memory_stats.stats.pgpgin 7388 1465019013\nstats.docker.registry.memory_stats.stats.pgpgout 4881 1465019013\nstats.docker.registry.memory_stats.stats.rss 7323648 1465019013\nstats.docker.registry.memory_stats.stats.rss_huge 2097152 1465019013\nstats.docker.registry.memory_stats.stats.total_active_anon 7340032 1465019013\nstats.docker.registry.memory_stats.stats.total_active_file 307200 1465019013\nstats.docker.registry.memory_stats.stats.total_cache 7131136 1465019013\nstats.docker.registry.memory_stats.stats.total_inactive_anon 0 1465019013\nstats.docker.registry.memory_stats.stats.total_inactive_file 6807552 1465019013\nstats.docker.registry.memory_stats.stats.total_mapped_file 1835008 1465019013\nstats.docker.registry.memory_stats.stats.total_pgfault 7622 1465019013\nstats.docker.registry.memory_stats.stats.total_pgmajfault 104 1465019013\nstats.docker.registry.memory_stats.stats.total_pgpgin 7388 1465019013\nstats.docker.registry.memory_stats.stats.total_pgpgout 4881 1465019013\nstats.docker.registry.memory_stats.stats.total_rss 7323648 1465019013\nstats.docker.registry.memory_stats.stats.total_rss_huge 2097152 1465019013\nstats.docker.registry.memory_stats.stats.total_unevictable 0 1465019013\nstats.docker.registry.memory_stats.stats.total_writeback 0 1465019013\nstats.docker.registry.memory_stats.stats.unevictable 0 1465019013\nstats.docker.registry.memory_stats.stats.writeback 0 1465019013\nstats.docker.registry.memory_stats.failcnt 0 1465019013\nstats.docker.registry.memory_stats.limit 12600770560 1465019013\nstats.docker.registry.networks.eth0.rx_bytes 110367 1465019013\nstats.docker.registry.networks.eth0.rx_packets 870 1465019013\nstats.docker.registry.networks.eth0.rx_errors 0 1465019013\nstats.docker.registry.networks.eth0.rx_dropped 0 1465019013\nstats.docker.registry.networks.eth0.tx_bytes 262883 1465019013\nstats.docker.registry.networks.eth0.tx_packets 836 1465019013\nstats.docker.registry.networks.eth0.tx_errors 0 1465019013\nstats.docker.registry.networks.eth0.tx_dropped 0 1465019013\n\n', children=[], properties={'priority': 0, 'content type': 'application/octet-stream', 'delivery mode': 1}>
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.cpu_usage.total_usage 0 1465019013
04/06/2016 07:43:34 :: invalid message line: stats.docker.registry.precpu_stats.cpu_usage.percpu_usage
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.cpu_usage.usage_in_kernelmode 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.cpu_usage.usage_in_usermode 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.system_cpu_usage 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.throttling_data.periods 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.throttling_data.throttled_periods 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.precpu_stats.throttling_data.throttled_time 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.cpu_usage.total_usage 10576789225 1465019013
04/06/2016 07:43:34 :: invalid message line: stats.docker.registry.cpu_stats.cpu_usage.percpu_usage 5380510422 2033927950 2470029525 692321328 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.cpu_usage.usage_in_kernelmode 410000000 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.cpu_usage.usage_in_usermode 3840000000 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.system_cpu_usage 1265626750000000 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.throttling_data.periods 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.throttling_data.throttled_periods 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.cpu_stats.throttling_data.throttled_time 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.usage 14454784 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.max_usage 31076352 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.active_anon 7340032 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.active_file 307200 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.cache 7131136 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.hierarchical_memory_limit 18446744073709551615 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.inactive_anon 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.inactive_file 6807552 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.mapped_file 1835008 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.pgfault 7622 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.pgmajfault 104 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.pgpgin 7388 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.pgpgout 4881 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.rss 7323648 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.rss_huge 2097152 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_active_anon 7340032 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_active_file 307200 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_cache 7131136 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_inactive_anon 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_inactive_file 6807552 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_mapped_file 1835008 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_pgfault 7622 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_pgmajfault 104 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_pgpgin 7388 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_pgpgout 4881 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_rss 7323648 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_rss_huge 2097152 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_unevictable 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.total_writeback 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.unevictable 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.stats.writeback 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.failcnt 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.memory_stats.limit 12600770560 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.rx_bytes 110367 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.rx_packets 870 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.rx_errors 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.rx_dropped 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.tx_bytes 262883 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.tx_packets 836 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.tx_errors 0 1465019013
04/06/2016 07:43:34 :: Metric posted: stats.docker.registry.networks.eth0.tx_dropped 0 1465019013
eheydrick commented 8 years ago

The problem is the cpu_usage.percpu_usage stat is returning multiple values but graphite expects a single value. Can you try the version at https://github.com/eheydrick/sensu-plugins-docker/blob/master/bin/metrics-docker-stats.rb to see if that fixes it for you. It filters out the invalid metrics lines.

celevra commented 8 years ago

sorry for the late answer. yes it resolved my Problem thank you!