Proxy Graphite requests through Descartes

[obfuscurity]

This has the benefit of avoiding CORS setup and troubleshooting. Unfortunately, it comes at the cost of latency thanks to blocking requests through Thin server. Casual tests revealed that a page of twelve (12) non-proxied graphs would load in approximately 400ms. The same graphs, loaded through the Thin server acting as a proxy, finished loading between 1400ms and 1800ms, with some taking as long as 2200ms.

In order for this to be a feasible design we'll have to consider moving to Unicorn for the extra threads. My initial stab at this was unsuccessful, unable to get Sequel connections loaded properly in the thread pool.

cc @gorsuch

[obfuscurity]

refs #106

[obfuscurity]

With this unicorn setup I'm seeing the same test run between 500-600ms with some outliers at 800ms. This seems like a reasonable compromise but I'm going to sleep on it before merging.

[gorsuch]

Looking :+1: to me!

[obfuscurity]

@gorsuch I'd like to get some subjective feedback from others as to whether this is A Good Thing (TM).

[rtyler]

Please don't throw a yak razor at me, but if you're concerned about speed and concurrency you might want to consider Puma instead of Unicorn.

Other than that, conceptually I see no problems with this, hitting the Graphite server directly has more landmines other than just CORS.

[ghost]

I've got dashboards that take 5--20 seconds to load; would prefer not to add additional latency if it's gonna cost ~25-50% extra.

[ghost]

Would it be prohibitively complex to support both code paths with an option in .env?

[jehiah]

FWIW in bitly/nsq the NSQ Admin UI has built in graphite support (for showing inline historical message queue metrics). By default it proxy's to graphite, but it has a command line argument to turn off that proxying, in which case you specify the base graphite URL and the NSQ admin interface just returns a 301 redirect to the right graphite render URL.

[clstokes]

It seems there would be greater value to the ops community as a whole by implementing CORS directly in Graphite rather than implementing a proxy in Descartes. Put another way, CORS in Graphite benefits (potentially) many projects as opposed to the proxy in Descartes only benefiting one project.

[obfuscurity]

@kingsbury I generally prefer One True Way (TM) of doing these sorts of things to ease supportability.

[obfuscurity]

@clstokes Graphite is not a webserver, so it cannot support CORS natively. It has to be handled at the webserver layer, e.g. Apache or Gunicorn.

[obfuscurity]

@rtyler @kingsbury I've tested a Puma branch and it seems to work well, with predictable results between 400-500ms.

This still doesn't address whether proxying is the correct way to approach this design, but I'm happy enough with Puma's results to justify replacing Unicorn with it for the memory savings.

[ghost]

I don't understand, @obfuscurity--graphite-web is a big ol django app, right? Surely it can talk about HTTP headers...

[clstokes]

I assumed this could be added to graphite-web as well.

[obfuscurity]

@kingsbury Django is a web framework, not a webserver. It still requires you to choose a webserver. http://en.wikipedia.org/wiki/Django_(web_framework)#Server_arrangements

[ghost]

Begging your pardon, I'm probably not thinking clearly, but an HTTP framework that can't emit HTTP headers seems unusual.

[ghost]

Yeah, here we go, @obfuscurity: https://github.com/ottoyiu/django-cors-headers/

[obfuscurity]

@kingsbury Good to know, but still not relevant to this discussion imo.

@clstokes was arguing that having CORS support natively in Graphite would be nice. I don't care whether it is or isn't; you can already use CORS with Graphite, via Apache or otherwise. The point of this PR is to determine whether, as a client, we should continue to use CORS for Graphite requests, or proxy through Descartes to bypass the requirements of CORS altogether.

[clstokes]

@obfuscurity I missed the point then, my mistake.

[obfuscurity]

@clstokes No worries, I learned something in the process. :smile_cat:

[obfuscurity]

After further tests I've decided not to merge this into master. The latency costs, while not obscene, have exceeded my comfort levels. As much as I'd love to see a better cross-origin solution, our CORS requirements are a small price to pay upfront for improved performance over the long haul.