Documentation updates for dev tools and a simple reverse proxy deployment

[jonnenauha]

I commented on a issue that this aspect of prometheus and promdash is not covered very well. I'm hoping new prometheus users like me will benefit from it written down somewhere. My biggest hurdle in even trying out prometheus for a long time was the lack of clear docs for how to restrict access to the monitoring.

As @juliusv noted in the issue the projects focus has not been on this kind of "small scale" setup you roll on your own, but for more experienced people and private clusters. I believe many small companies might benefit in what I wrote down as its fairly simple to setup. This is for small deployments as everything is running on a single machine and means you cant easily add more prometheus servers. You can by doing more reverse proxy magic and esp. once the basic auth in the server URL is implemented.

I can also look into making a more in detail doc page into https://github.com/prometheus/docs with all details and configs for HAProxy as that is what I know how to use now. If you think its a good idea.

If the dev tool install part is too much detail in your opinion, I made it a separate commit so its easy to revert.

Feel free to comment on any typos or bad wording, I'll fix them up.

[jonnenauha]

Hmm, running this documented setup once again in our cloud and does not seem to work again. dash cant still poll prom. Hold on the potential merge, I'll need to still sort this out :P Damn...

Edit: Right I was running prometheus without docker when testing earlier. Seems that running it in that breaks it. Or then there is some special breakage in the ruby/js side for localhost and 127.0.0.1, they do not seem to do anything. Any ideas from the devs, have you ran prom + dash locally and used http://localhost:9090/ as the endpoint?

[jonnenauha]

Ok, so I guess I've missed the idea of promdash by a big margin. Looking at the HAProxy logs the poll requests are actually coming from my IP not from the server. I was under the impression that prom polls the metrics, I cant say why but just assumed that would be more efficient :) (one querying instead of all clients who have the page open). But i guess then sending them to the client would be double the work and the bottle neck could be promdash instead.

Alright, so this pull request cant be merged before the http://user:pass@host:9090/ issue is resolved. After that I can do a slight tweak to this and it will actually be true :)

[jonnenauha]

Now that I understood this I whitelisted our companys office IP in HAProxy and it started working. So this is enough for my use cases until that basic auth in the URL is implemented. If you think this would be good doc if I change it to note the whitelisting of "trusted srcs" I can update the docs too, and then update again with basic auth stuff.

[juliusv]

Cool, thanks for doing this! More detailed comments coming now...

[jonnenauha]

@juliusv thanks for the help. Hope I covered them all.

I'm thinking I should modify this a bit to actually document the single domain work around from https://github.com/prometheus/promdash/issues/341#issuecomment-74425398

I did this last night with HAProxy and it works well. I was able to expose /prom/api > :9090/api /prom/graph > :9090/graph and /static > :9090/static (this cant be prefixed with /prom as the /graph page points to it relatively) by doing path rewrites to the request. That graph part is missing from the work around, it proved to be sometimes valuable to also have access to the graph at prometheus.

However this increases the complexity a bit and will make the examples longer and more complex to understand. I think I could just doc the /api/ for both nginx and haproxy?

This might be better than waiting for the basic auth for prom server endpoint fix. Would get something visible now, and if you want it could be changed to two domains later once the fix is in.

[juliusv]

Yep, I'd be all for describing the single-domain variant, as that's the only thing that is working at the moment (and nobody is actively working on a fix). Maybe just route all of /prom to Prometheus, not just certain whitelisted endpoints. It's all behind auth anyways, right?

[jonnenauha]

Can you point me to to the right file/lines where I could take a peek at the basic auth problem. I have a lot of experience with js and have used angular in a few projects. Could try to figure out the basic auth bug.

/prom will work fine for the API. If you want /graph to render properly it will be a bit more complex. So I agree that the single prefix would be good, makes the explanation a lot simpler.

I really wish prometheus would also allow to modify the root prefix that it mounts all its handlers to. similar to what your PROMDASH_PATH_PREFIX does.

If it did, you could just run it with /prom and no additional "hacks" with the reverse proxy would be required. Would fix the graph/static problem. I wonder if this could be done in the main repo, maybe I should make a issue there explaining the situation?

I try to update this PR in the upcoming days to something that works today.

[stuartnelson3]

my idea regarding basic auth was written here: https://github.com/prometheus/promdash/issues/341#issuecomment-131769919

[juliusv]

@jonnenauha Cool! Yeah, @stuartnelson3 already pointed to how to tackle the auth problem in general.

As for making Prometheus mount all its handlers on a path prefix, that already exists. But it's a command-line flag in the Prometheus server:

./prometheus  -h | grep -A3 -i external
   -web.external-url 
      The URL under which Prometheus is externally reachable (for 
      example, if Prometheus is served via a reverse proxy). Used for 
      generating relative and absolute links back to Prometheus itself. If 
      omitted, relevant URL components will be derived automatically.

[jonnenauha]

Ok, surely read that but I failed to understand you can also give a path portion. Thought its just be base URL. How does that work exactly if the server is listening to e.g. localhost:9090. I suppose you parse the path part and prefix all handlers with it?

So would -web.external-url "http://dash.example.com/prom" result in it mounting root (index page) to localhost:9090/prom/ and graph to localhost:9090/prom/graph?

This is what I was looking for.

Edit: My idea was that if this is possible, it would make all the reverse proxy on the fly path modifications unnecessary when both are in the same domain. Now if you want all pages to work well, you need to do some more complex dynamic path rewrites in the proxy.

[juliusv]

Yeah, that's how it works - Prometheus will mount all its endpoints under /prom in your example, and will also adjust all links pointing to itself to include that path prefix.

[jonnenauha]

Wow, thats great. Maybe you could update the doc string in the -h to reflect this better, something like

If the URL has a path portion, it will be used to prefix all the HTTP handlers.

You can probably work it more concisely but I had a hard time understand it does that from the current doc string, maybe I'm not the only one. Just a suggestion.

[juliusv]

@jonnenauha Good point. The path prefix features both in Prometheus and PromDash were introduced by external contributors who needed them, so they haven't always gotten the full attention they deserved. Here's a PR for adding your suggestion: https://github.com/prometheus/prometheus/pull/1089

[jonnenauha]

Alright, I've changed the documentation to use the single domain approach and tested in our production this actually work :) For good measure I added minimal HAProxy and Nginx (untested, modified from https://github.com/prometheus/promdash/issues/341#issuecomment-74425398) configs.

If you think this is too much detail, takes too much space or is up to the user to figure out I made it a separate commit so its easy to revert.

[juliusv]

Ok yeah, it's getting long, but I guess it's still ok to have it in the README.md for now. We might move it to a separate place at some point and link to it. Giving it a closer look now.

[juliusv]

Sorry for being a bit slow on this PR. Hmm, we used the wiki once, but decided to close it because we already have two places for documentation: the README.mds and prometheus.io (which includes blog posts). I think long-term we will want a separate "Howtos" section on prometheus.io or something.

For now, it's ok as it is now, after addressing https://github.com/prometheus/promdash/pull/449/files#r39730036 and https://github.com/prometheus/promdash/pull/449/files#r39730005.

Thanks!

[juliusv]

(and then please squash your commits into a single one before merging)

[jonnenauha]

Yeah, I'm in no rush here, just trying to help :)

I've used git for a very long time but have not had to use squash before. Lets see if i cant get this done without screwing things up!

[jonnenauha]

Got a bit nervous about the --force push but looks like it did what you wanted. As this is a feature branch ofc force was fine, still :)

[juliusv]

@jonnenauha Ah, better now, thanks :) What about this last comment from 6d ago? https://github.com/prometheus/promdash/pull/449/files#r39729955

[jonnenauha]

I did not do that on my own setup. Will make it a little ugly and you have to remember to postfix your url with /prom when you want to go look at your metrics :P I guess one could do a redirect from the root to /prom in the reverse proxy by default. Seems kind of ugly but I see your point and people might make a prom named dashboard.

I'll change it.

[juliusv]

@jonnenauha Thanks again for sticking with this :) Merging!