search

vimeo / graph-explorer

A graphite dashboard powered by structured metrics
http://vimeo.github.io/graph-explorer/
Apache License 2.0
1.06k stars 93 forks source link

Update_metrics.py stuck after listing metrics #78

Closed rickstok closed 10 years ago

rickstok commented 10 years ago

update_metrics.py is hanging after list_metrics, What could cause the problem, is it some python modules that are not loaded?

oot@mon-gui:/opt/graph-explorer# ./update_metrics.py 
2013-12-18 11:20:18,500 - update_metrics - INFO - fetching/saving metrics from graphite...
2013-12-18 11:20:18,590 - update_metrics - INFO - generating structured metrics data...
2013-12-18 11:20:18,590 - update_metrics - DEBUG - loading metrics
2013-12-18 11:20:18,595 - update_metrics - DEBUG - removing outdated targets
2013-12-18 11:20:18,596 - update_metrics - DEBUG - making sure index exists..
No handlers could be found for logger "elasticsearch"
2013-12-18 11:20:18,599 - update_metrics - DEBUG - making sure shard is started..
2013-12-18 11:20:18,600 - update_metrics - DEBUG - shard is ready!
2013-12-18 11:20:18,601 - update_metrics - DEBUG - making sure index exists..
2013-12-18 11:20:18,603 - update_metrics - DEBUG - making sure shard is started..
2013-12-18 11:20:18,604 - update_metrics - DEBUG - shard is ready!
2013-12-18 11:20:18,606 - update_metrics - DEBUG - removed 0 metrics from elasticsearch
2013-12-18 11:20:18,606 - update_metrics - DEBUG - updating targets
2013-12-18 11:20:18,607 - update_metrics - DEBUG - list_metrics with 24 plugins and 4340 metrics
2013-12-18 11:20:18,887 - update_metrics - DEBUG -          plugin name   metrics upgrade ok  metrics upgrade bad      metrics ignored
2013-12-18 11:20:18,887 - update_metrics - DEBUG -        native_proto2                    0                    0                    0
2013-12-18 11:20:18,887 - update_metrics - DEBUG -               carbon                   13                    0                    0
2013-12-18 11:20:18,887 - update_metrics - DEBUG -             collectd                    0                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG -                  cpu                  420                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG - diamondcollectortime                    0                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG -            diskspace                  234                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG -             filestat                    0                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG -               iostat                 2376                    0                    0
2013-12-18 11:20:18,888 - update_metrics - DEBUG -                 load                   90                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG -               memory                  252                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG -              network                  448                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG -             sockstat                  144                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG -               statsd                    0                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG -                swift                    0                    0                    0
2013-12-18 11:20:18,889 - update_metrics - DEBUG - swift_object_auditor                    0                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -  swift_object_server                    0                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -   swift_proxy_server                    0                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -       swift_tempauth                    0                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -                  tcp                  288                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -                  udp                    0                    0                    0
2013-12-18 11:20:18,890 - update_metrics - DEBUG -               vmstat                   72                    0                    0
2013-12-18 11:20:18,891 - update_metrics - DEBUG -     catchall_diamond                    0                    0                    0
2013-12-18 11:20:18,891 - update_metrics - DEBUG -      catchall_statsd                    0                    0                    0
2013-12-18 11:20:18,891 - update_metrics - DEBUG -             catchall                    3                    0                    0
2013-12-18 11:20:18,891 - update_metrics - DEBUG - making sure index exists..
2013-12-18 11:20:18,893 - update_metrics - DEBUG - making sure shard is started..
2013-12-18 11:20:18,894 - update_metrics - DEBUG - shard is ready!
^CTraceback (most recent call last):
  File "./update_metrics.py", line 36, in <module>
    backend.update_data(s_metrics)
  File "/opt/graph-explorer/backend.py", line 76, in update_data
    s_metrics.update_targets(metrics)
  File "/opt/graph-explorer/structured_metrics/__init__.py", line 337, in update_targets
    self.es_bulk(bulk_list)
  File "/opt/graph-explorer/structured_metrics/__init__.py", line 275, in es_bulk
    self.es.bulk(index='graphite_metrics', doc_type='metric', body=body)
  File "/opt/graph-explorer/structured_metrics/elasticsearch-py/elasticsearch/client/utils.py", line 70, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/opt/graph-explorer/structured_metrics/elasticsearch-py/elasticsearch/client/__init__.py", line 522, in bulk
    params=params, body=self._bulk_body(body))
  File "/opt/graph-explorer/structured_metrics/elasticsearch-py/elasticsearch/transport.py", line 223, in perform_request
    status, raw_data = connection.perform_request(method, url, params, body, ignore=ignore)
  File "/opt/graph-explorer/structured_metrics/elasticsearch-py/elasticsearch/connection/http_urllib3.py", line 44, in perform_request
    response = self.pool.urlopen(method, url, body, **kw)
  File "/opt/graph-explorer/structured_metrics/urllib3/urllib3/connectionpool.py", line 428, in urlopen
    body=body, headers=headers)
  File "/opt/graph-explorer/structured_metrics/urllib3/urllib3/connectionpool.py", line 288, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
Dieterbe commented 10 years ago

so structured_metrics is in update_targets(), it does assure_index() properly and then gets stuck in self.es_bulk(), https://github.com/vimeo/graph-explorer/blob/master/structured_metrics/__init__.py#L275

i don't think i can do anything about this, this seems to be either a bug in the official elasticsearch-py library (maybe try opening a ticket with them) or maybe you had a network/ ES connectivity problem?