search

prometheus-community / PushProx

Proxy to allow Prometheus to scrape through NAT etc.
Apache License 2.0
705 stars 132 forks source link

"malformed HTTP request \"\"" #118

Open scythargon opened 2 years ago

scythargon commented 2 years ago

Hi! I'm having troubles running PushProx on my staging server.

The most informative error I see is from proxy client receiving bad request:

$ ./pushprox-client --proxy-url=http://prom-proxy-stg.***.com/ --fqdn=`hostname` --metrics-addr=":9368"
level=info ts=2021-09-22T10:26:52.905Z caller=main.go:245 msg="URL and FQDN info" proxy_url=http://prom-proxy-stg.***.com/ fqdn=argon-u1910
level=error ts=2021-09-22T10:27:53.467Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""
level=error ts=2021-09-22T10:28:55.319Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""

And then probably because of that pushprox-proxy fails to retrieve data from the client:

caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=argon-u1910
caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"

Some more details:

The client gets successfully registered at proxy/clients page: [{"targets":["argon-u1910"],"labels":null}]

And Prometheus is able to see that client live, but fails to scrape.

Prometheus config files and dashboard screenshots: Selection_696

If I run proxy-server locally - everything is fine. So the problem is probably somewhere in the stack on which I'm trying to host it on cloud my servers. Should there be some additional ports open or something? Might the problem be in some http/https redirect where the request body is lost?

Any help will be much appreciated!

Thank you!

scythargon commented 2 years ago

The problem lies somewhere in reverse proxy configuration which is placed in front of PushProx. I believe there is one at my cloud service provider side which I can't really configure. Then I've tried to install PushProx in my AWS EC2 instance behind Nginx - this didn't work either - was receiving err="malformed HTTP request \"<html>\" error instead. But it works it if just reach to PushProx listening port directly. (Had to make in available in AWS firewall). So if you can point me on what nginx configuration would be suitable here - that would be nice:) Thanks!

bingtw commented 2 years ago

I got the same issue when my PushProx service is located behind AWS ALB. (client <-> AWS ALB <-> server) But it works properly when I use public IP address of my PushProx service in client. (client <-> server)

nitinakash1989 commented 1 year ago

I am also facing the same issue

ipsecguy commented 1 year ago

Same here on local infra. I have to encrypt the traffic (Security will reject otherwise, and probably rightfully so).

Same error happens regardless whether I specify a valid URL (http://server1:9100/metrics for node exporter, an invaldi URL or an invalid path. So I suspect the proxy client does not even try to open a connection or the port/path is lost on the way even though it is displayed correctly in the "Got scrape request" log on the client.

menvol3 commented 3 months ago

I faced the same issue. Does anyone know how to resolve it?