search

obfuscurity / descartes

Introspective dashboard for Graphite
MIT License
502 stars 65 forks source link

Metrics cache #54

Closed obfuscurity closed 11 years ago

obfuscurity commented 12 years ago

Cache the output of /metrics/index.json somewhere we can use it for "realtime" queries. Redis would probably be a good place for this.

obfuscurity commented 12 years ago

Initial tests against a moderately populated Graphite (~35k unique metrics, 200KB /metrics/index.json output) show that it takes ~1.5 seconds to render and download. It takes almost the same amount of time to pull this from hosted RedisToGo or Heroku Postgres services. For now we'll just move forward without the cache, download in the background on page-load, and revisit this when/if it becomes necessary.

obfuscurity commented 11 years ago

It became obvious that larger Graphite installations won't scale for this approach. A server with 60K+ unique metrics results in an index.json of almost 5MB taking 15 seconds to download. Much of this is time spent compressing (in our case, with gunicorn) but the benchmark is almost the same unzipped. Web caching alleviates some of this, but it's still not a good approach.

obfuscurity commented 11 years ago

:cake: