search

vimeo / graphite-influxdb

An influxdb backend for Graphite-web and graphite-api
Apache License 2.0
198 stars 39 forks source link

the Installation needs graphite-api even when install on graphite-web #10

Closed toni-moreno closed 10 years ago

toni-moreno commented 10 years ago

We would like to install graphite-web on front of a influxdb database but graphite-influxdb needs graphite-api !!

can I configure on with graphite-web without graphite-api ?

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 101, in get_response
    request.path_info)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 209, in resolve
    return ResolverMatch(self.callback, args, kwargs, self.name)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 216, in callback
    self._callback = get_callable(self._callback_str)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 27, in wrapper
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 92, in get_callable
    lookup_view = getattr(import_module(mod_name), func_name)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 21, in <module>
    from graphite.storage import STORE
  File "/opt/graphite/webapp/graphite/storage.py", line 165, in <module>
    STORE = Store()
  File "/opt/graphite/webapp/graphite/storage.py", line 27, in __init__
    for finder_path in settings.STORAGE_FINDERS]
  File "/opt/graphite/webapp/graphite/storage.py", line 20, in get_finder
    return getattr(module, class_name)()
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb-0.2-py2.7.egg/graphite_influxdb.py", line 223, in __init__
    from graphite_api.app import app
ImportError: No module named graphite_api.app
Dieterbe commented 10 years ago

ah yes, that's for the new caching feature. we'll need to do a try/except here and import the cache from graphite-web if it fails. not sure yet how to import cache from graphite-web yet, as i don't use graphite-web anymore.

toni-moreno commented 10 years ago

excuse me for my ignorance, what caching feature are you referring for ?

anyway I tried to install graphite-api ( with pip install) we got the following error.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 101, in get_response
    request.path_info)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 209, in resolve
    return ResolverMatch(self.callback, args, kwargs, self.name)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 216, in callback
    self._callback = get_callable(self._callback_str)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 27, in wrapper
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 101, in get_callable
    (lookup_view, mod_name))
ViewDoesNotExist: Could not import graphite.metrics.views.find_view. View does not exist in module graphite.metrics.views.

I'm really interested in testing influxdb with graphite-web, if you can guide me and is not a very complex task we can help you to make it run also on graphite-web

Dieterbe commented 10 years ago

graphite-influxdb needs a cache to maintain some datastructures and speed up stuff (see mentions of cache in the readme). graphite-api exposes the cache object, which we import. i know graphite-web has a similar cache object somewhere. in case the former fails, we should import the latter.

not sure what that error is. maybe graphite-api didn't install well (you can try executing the import statements in a python console to verify). can't look into it right now (i'm on vacation :))

toni-moreno commented 10 years ago

after adding a try: except: 

        try:
           from graphite_api.app import app
           self.cache = app.cache
        except ImportError:
           self.cache = None

Now statsd is the problem:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
    matches = list( STORE.find(query, fromTime, untilTime, local=local_only) )
  File "/opt/graphite/webapp/graphite/storage.py", line 47, in find
    for node in finder.find_nodes(query):
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb-0.2-py2.7.egg/graphite_influxdb.py", line 339, in find_nodes
    with statsd.timer('service=graphite-api.action=yield_nodes.target_type=gauge.unit=ms.what=query_duration'):
AttributeError: 'NoneType' object has no attribute 'timer'

I've redefined Nullstatsd

def NullStatsd():

by

class NullStatsd():

and now the exception has changed :

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
    matches = list( STORE.find(query, fromTime, untilTime, local=local_only) )
  File "/opt/graphite/webapp/graphite/storage.py", line 47, in find
    for node in finder.find_nodes(query):
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb-0.2-py2.7.egg/graphite_influxdb.py", line 320, in find_nodes
    with statsd.timer('service=graphite-api.action=yield_nodes.target_type=gauge.unit=ms.what=query_duration'):
AttributeError: __exit__

I'm not able to redefine a fully working NullStat() class. Can you help me when you return from vacation !!.

thank you very much.

Dieterbe commented 10 years ago

can you try putting this method in the NullStatsd class:

def __enter__(self):
    return self
def __exit__(self, type, value, traceback):
    pass

that should work

Dieterbe commented 10 years ago

Hi Toni, i just pushed the fixes for importing graphite-web/django caching as well as the NullStatsd fixes.

So now it should work. let me know how it goes and feel free to reopen if you keep getting issues

toni-moreno commented 10 years ago

Hi. I will test as soon as I come back home. I'm just having a vacation days and far of home.

Thank you very much.

toni-moreno commented 10 years ago

Hi @Dieterbe I'm back and I've tested current master version. But is still crashing on the NullStatd class exit method.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
    matches = list( STORE.find(query, fromTime, untilTime, local=local_only) )
  File "/opt/graphite/webapp/graphite/storage.py", line 47, in find
    for node in finder.find_nodes(query):
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb-0.2-py2.7.egg/graphite_influxdb.py", line 347, in find_nodes
    with statsd.timer('service=graphite-api.action=yield_nodes.target_type=gauge.unit=ms.what=query_duration'):
AttributeError: __exit__

Any way to bypass it?, perhaps a good way could be a graphite-influxdb without statd support. ?

Dieterbe commented 10 years ago

the fix for that bug was merged yesterday, should work fine now :)

toni-moreno commented 10 years ago

Sorry

With the last version. It is still crashing... :(

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
    matches = list( STORE.find(query, fromTime, untilTime, local=local_only) )
  File "/opt/graphite/webapp/
graphite/storage.py", line 47, in find
    for node in finder.find_nodes(query):
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb.py", line 348, in find_nodes
    for (name, res) in self.get_leaves(query):
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb.py", line 307, in get_leaves
    series, regex = self.assure_series(query)
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb.py", line 278, in assure_series
    raise Exception("series not in cache. please run maintain_cache.py")
Exception: series not in cache. please run maintain_cache.py
Dieterbe commented 10 years ago

are you running the included maintain_cache.py script?

toni-moreno commented 10 years ago

Hi @Dieterbe.

We have executed it but remember we are trying to install graphite-influxdb into graphite-web so we have not installed graphite-api.

The following error is shown

python /root/graphite-influxdb/bin/maintain_cache.py
Traceback (most recent call last):
  File "/usr/local/bin/maintain_cache.py", line 45, in <module>
    with open(config_file) as f:
IOError: [Errno 2] No such file or directory: '/etc/graphite-api.yaml'

How can we bypass this cache framework ?

Thank you very much!

Dieterbe commented 10 years ago

I removed maintain_cache.py, all that stuff is obsolete now. (see latest commits)

toni-moreno commented 10 years ago

OK!!! much better!!

now the error seems on attribute config to the finder init .

  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 101, in get_response
    request.path_info)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 209, in resolve
    return ResolverMatch(self.callback, args, kwargs, self.name)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 216, in callback
    self._callback = get_callable(self._callback_str)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 27, in wrapper
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 92, in get_callable
    lookup_view = getattr(import_module(mod_name), func_name)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 21, in <module>
    from graphite.storage import STORE
  File "/opt/graphite/webapp/graphite/storage.py", line 165, in <module>
    STORE = Store()
  File "/opt/graphite/webapp/graphite/storage.py", line 27, in __init__
    for finder_path in settings.STORAGE_FINDERS]
  File "/opt/graphite/webapp/graphite/storage.py", line 20, in get_finder
    return getattr(module, class_name)()
  File "/usr/local/lib/python2.7/dist-packages/graphite_influxdb.py", line 209, in __init__
    self.schemas = [(re.compile(patt), step) for (patt, step) in config['influxdb']['schema']]
TypeError: 'NoneType' object has no attribute '__getitem__'

We've put a trace to get current data contained in the config object.

f.open("/tmp/influxdb.log") f.write(repr(config)) f.close()

The file /tmp/influxdb.log has only the string "None"

Dieterbe commented 10 years ago

yes, you need to put the schemas in your config, see the readme

toni-moreno commented 10 years ago

yes , we had 

STORAGE_FINDERS = (
        'graphite_influxdb.InfluxdbFinder',
)
INFLUXDB_HOST = "192.168.150.113"
INFLUXDB_PORT = 8086
INFLUXDB_USER = "graphite"
INFLUXDB_PASS = "graphite"
INFLUXDB_DB =  "graphitedb"
INFLUXDB_SCHEMA = [
    ('', 60),
    ('high-res-metrics', 10)
]

At local_settings.py when we got last showed stack trace error.

Dieterbe commented 10 years ago

should be fixed via 506d4f98881b32e831a9c4fb1e1508d0cae3bf15

toni-moreno commented 10 years ago

OK ! it seems to connect ok . So I will consider this issue finally as closed and I will open a new one

https://github.com/vimeo/graphite-influxdb/issues/22

Thank you very much.