Add option for clients to forward requests from the host IP to 127.0.0.1

der-ali commented 2 months ago

If some services listens to localhost instead of the node's FQDN/public ip, it's currently not possible for the client to scrape their metrics.

This feature is already implements in the Rancher's PushProx fork

peekjef72 commented 3 weeks ago

Maybe it is not a problem anymore but I can't reproduce your use case.

Can you specify the config please ?

Here is a try:

start proxy (listen on :8080)

$ ./pushprox_proxy &
ts=2024-06-01T15:22:35.877Z caller=main.go:373 level=info msg=Listening address=:8080

start client identified by localhost and connected to proxy

$./pushprox-client --fqdn=localhost --proxy-url=http://192.168.0.196:8080 &
ts=2024-06-01T15:25:07.579Z caller=main.go:243 level=info msg="URL and FQDN info" proxy_url=http://192.168.0.196:8080/ fqdn=localhost

try to scrap an exporter (httpapi_export bind on localhost

curl --proxy 127.0.0.1:8080 -H "X-Prometheus-Scrape-Timeout-Seconds: 500" http://localhost:9321
<html>
      <head>
        <title>Prometheus httpapi_exporter</title>
        <style type="text/css">
          body { margin: 0; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 1.42857143; color: #333; background-color: #fff; }
          .navbar { display: flex; background-color: #222; margin: 0; border-width: 0 0 1px; border-style: solid; border-color: #080808; }
          .navbar > * { margin: 0; padding: 15px; }
          .navbar * { line-height: 20px; color: #9d9d9d; }
          .navbar a { text-decoration: none; }
          .navbar a:hover, .navbar a:focus { color: #fff; }
          .navbar-header { font-size: 18px; }
          body > * { margin: 15px; padding: 0; }
          pre { padding: 10px; font-size: 13px; background-color: #f5f5f5; border: 1px solid #ccc; }
          h1, h2 { font-weight: 500; }
          a { color: #337ab7; }
          a:hover, a:focus { color: #23527c; }
          table { border: 1px solid #edd2e6; border-collapse: collapse; margin-bottom: 1rem; width: 80%; }
          tr { border: 1px solid #edd2e6; padding: 0.3rem; text-align: left; width: 35%; }
          th { border: 1px solid #edd2e6; padding: 0.3rem; }
          td { border: 1px solid #edd2e6; padding: 0.3rem; }
          .odd { background-color: rgba(0,0,0,.05); }
        </style>
      </head>
      <body>
        <div class="navbar">
          <div class="navbar-header"><a href="/">Prometheus httpapi_exporter</a></div>
          <div><a href="/healthz">Health</a></div>
          <div><a href="/metrics">Metrics</a></div>
          <div><a href="/config">Configuration</a></div>
          <div><a href="/targets">Targets</a></div>
          <div><a href="/status">Status</a></div>
          <div><a href="/debug/pprof">Profiling</a></div>
          <div><a href="/httpapi_exporter_metrics">Exporter Metrics</a></div>
          <div><a href="https://github.com/peekjef72/httpapi_exporter#readme">Help</a></div>
        </div>
        <p>This is a <a href="https://github.com/peekjef72/httpapi_exporter#readme">Prometheus httpapi_exporter</a> instance.
        You are probably looking for its <a href="/metrics">metrics</a> handler.</p>
      </body>
      </html>

and logs form both proxy and client

ts=2024-06-01T15:38:45.694Z caller=coordinator.go:122 level=info msg=DoScrape scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 url=http://localhost:9321/
ts=2024-06-01T15:38:45.694Z caller=main.go:208 level=info msg="Responded to /poll" url=http://localhost:9321/ scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.695Z caller=main.go:206 level=info msg="Got scrape request" scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 url=http://localhost:9321/
ts=2024-06-01T15:38:45.695Z caller=coordinator.go:143 level=info msg=WaitForScrapeInstruction fqdn=localhost
ts=2024-06-01T15:38:45.696Z caller=main.go:140 level=info scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 msg="Retrieved scrape response"
ts=2024-06-01T15:38:45.697Z caller=main.go:189 level=info msg="Got /push" scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.697Z caller=coordinator.go:175 level=info msg=ScrapeResult scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.697Z caller=main.go:146 level=info scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 msg="Pushed scrape result"

der-ali commented 3 weeks ago

Thanks a lot @peekjef72 :) I currently use the pushprox-client from the Rancher project, but I'm considering switching to the upstream version. I'll be testing it out in the coming days.

prometheus-community / PushProx

Add option for clients to forward requests from the host IP to 127.0.0.1 #173