vimeo / graph-explorer

A graphite dashboard powered by structured metrics
http://vimeo.github.io/graph-explorer/
Apache License 2.0
1.06k stars 93 forks source link

elasticsearch timeout #56

Closed Dieterbe closed 10 years ago

Dieterbe commented 10 years ago
127.0.0.1 - - [30/Oct/2013:16:36:59 -0400] "POST /graphs/ HTTP/1.1" 500 748 "http://localhost:8080/index/tc%20wait%20vimeo-df" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49506)
Traceback (most recent call last):
  File "/home/dieter/workspaces/eclipse/graph-explorer/paste/httpserver.py", line 1069, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Traceback (most recent call last):
  File "/home/dieter/workspaces/eclipse/graph-explorer/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/home/dieter/workspaces/eclipse/graph-explorer/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/home/dieter/workspaces/eclipse/graph-explorer/app.py", line 326, in graphs
    return handle_graphs(query, False)
  File "/home/dieter/workspaces/eclipse/graph-explorer/app.py", line 348, in handle_graphs
    return render_graphs(query, deps=deps)
  File "/home/dieter/workspaces/eclipse/graph-explorer/app.py", line 398, in render_graphs
    (query, targets_matching) = s_metrics.matching(query)
  File "/home/dieter/workspaces/eclipse/graph-explorer/structured_metrics/__init__.py", line 313, in matching
    self.assure_index()
  File "/home/dieter/workspaces/eclipse/graph-explorer/structured_metrics/__init__.py", line 174, in assure_index
    self.es.post('graphite_metrics', data=body)
  File "/home/dieter/workspaces/eclipse/graph-explorer/structured_metrics/rawes/rawes/elastic.py", line 61, in post
    return self.request('post', path, **kwargs)
  File "/home/dieter/workspaces/eclipse/graph-explorer/structured_metrics/rawes/rawes/elastic.py", line 83, in request
    return self.connection.request(method, new_path, **kwargs)
  File "/home/dieter/workspaces/eclipse/graph-explorer/structured_metrics/rawes/rawes/http_connection.py", line 40, in request
    response = self.session.request(method, "/".join((self.url, path)), **args)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 360, in send
    raise Timeout(e)
Timeout: HTTPConnectionPool(host='graphiteserver', port=9202): Read timed out. (read timeout=30)

might be related / same issue as #45

Dieterbe commented 10 years ago

it looks like rawes' Elastic class keeps a connection to the ES, but when it times out, there doesn't seem to be an easy way to reconnect (automatic or manual), besides creating a new rawes object.

Dieterbe commented 10 years ago

reported at humangeo/rawes/issues/19

Dieterbe commented 10 years ago

gonna assume ebb5b49dee70a96552cdcd035f4452838d8551c5 fixed this, reopen if not