search

PaloAltoNetworks / minemeld

Main MineMeld documentation repo
379 stars 64 forks source link

Error retrieving minemeld metrics: Internal Server Error #36

Closed nkovacne closed 5 years ago

nkovacne commented 5 years ago

Just cloned the master branch of this project on a CentOS 7 machine, followed the steps to install it on that OS, restored some data from a previous version we were handling and each time I refresh the Dashboard I get a few Python exceptions:


[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] RPC sent to @mbus:master:rpc for method status
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /status/minemeld?_=1557994429 HTTP/1.0" 200 8245 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [INFO] redis connection pool: in use: 0 available: 1
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /extensions?_=1557994429 HTTP/1.0" 200 19 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /aaa/users/current?_=1557994429 HTTP/1.0" 200 71 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /supervisor?_=1557994429 HTTP/1.0" 200 594 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 2 available: 4
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 3 available: 3
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /status/info?_=1557994429 HTTP/1.0" 200 65 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld/miners [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 127, in get_node_type_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld/miners?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /config/running?_=1557994429 HTTP/1.0" 200 4732 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /prototype?_=1557994429 HTTP/1.0" 200 261753 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 2 available: 4
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 170, in get_global_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld/outputs [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 127, in get_node_type_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld/outputs?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:13:51 +0100] "GET /prototype?_=1557994430 HTTP/1.0" 200 261753 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:14:00 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:14:00 WEST] [19313] [DEBUG] RPC sent to @mbus:master:rpc for method status
127.0.0.1 - - [16/May/2019:09:14:00 +0100] "GET /status/minemeld?_=1557994294 HTTP/1.0" 200 8245 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"
[2019-05-16 09:14:01 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:14:01 +0100] "GET /supervisor?_=1557994295 HTTP/1.0" 200 594 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"```
nkovacne commented 5 years ago

Problem of ours, overlapping our own collectd.