NGINX Proxy configuration ignores query params

[lokiwins]

• Which Faktory package and version? 1.5.0
• Which Faktory worker package and version? N/A Nginx Configuration followed what was outlined in: https://github.com/contribsys/faktory/blob/master/webui/web.go#L154

Config:

server {
    listen 80;
    server_name engineering.{{ .Values.domain }};

    location = /ping {
      access_log off;
      return 200 "pong/n";
    }

    location = /50x_error.html {
      root /var/www/html;
      internal;
    }

    location /faktory {
      proxy_set_header X-Script-Name /faktory;
      proxy_pass http://faktory;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Scheme $scheme;
      proxy_set_header X-Real-IP $remote_addr;
    }
 }

http://faktory is just a headless service in kubernetes that hits the faktory pod on the web port 7420

Are you using an old version? Slightly. 1.5.0 vs 1.5.1 Have you checked the changelogs to see if your issue has been fixed in a later version? Yes

The main issue is most of it works except for the pagination. For example hitting /faktory/retries works fine, but hitting /faktory/retries?page=2 ends up just going to the home page.

Some additional detail: This seems to only be an issue when doing nginx proxy. If I hit the web ui directly it works just fine.

Example: With the above nginx configured for localhost domain and faktory running locally http://engineering.localhost/faktory/retries?page=2 Does not work - Proxy involved http://localhost:7420/retries?page=2 Works fine - No proxy

Be happy to provide any additional details

Thanks!

[mperham]

Take a look, let me know what you come up with.

https://stackoverflow.com/questions/8130692/how-can-query-string-parameters-be-forwarded-through-a-proxy-pass-with-nginx

[lokiwins]

Take a look, let me know what you come up with.

https://stackoverflow.com/questions/8130692/how-can-query-string-parameters-be-forwarded-through-a-proxy-pass-with-nginx

I went through all of the configurations and even tried using rewrite...break They still end up routing to the home page when a query param is passed.

proxy_pass should keep the query params without any fancy regex matching and then force passing them.

In the nginx logs I can see the request with query params

172.21.0.1 - - [04/May/2021:20:34:46 +0000] "GET /faktory/retries?page=2 HTTP/1.1" 200 8069 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"

And then on the faktory logs side I see it as well with the query params

I 2021-05-04T20:34:46.396Z GET /retries?page=2 2.560234ms

It just ends up loading the home page

[mperham]

I don't understand how it can log the query params but not see them. What is the definition of faktory in proxy_pass http://faktory;?

[lokiwins]

I don't understand how it can log the query params but not see them. What is the definition of faktory in proxy_pass http://faktory;?

Locally its defined inside nginx

upstream faktory {
    server host.docker.internal:7420 fail_timeout=0;
  }

Then when its deployed its a Kubernetes service that targets the faktory server pod on 7420

[mperham]

I'm stumped. We need to debug this but I don't have a local nginx setup to test with so it'll take some time to investigate. If you learn anything more, let me know.

[lokiwins]

I'm stumped. We need to debug this but I don't have a local nginx setup to test with so it'll take some time to investigate. If you learn anything more, let me know.

Tried some bogus query params and noticed that when its behind a proxy and ? is passed it results in that same behavior. Doesn't seem to be limited to just /retries. If I do /queues? it does the same thing. Works fine when not behind the proxy.

[lokiwins]

I was able to get a look at the difference between the request via proxy vs no proxy.

Request with Proxy:
&{GET /retries%3Fpage=2?page=2 HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9] Connection:[close] Cookie:[csrf_token=mIlWhjEHfh5C/IyXl2V8wsOtJCx5clacBMLO1UmvNK8=] Sec-Ch-Ua:[" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36] X-Script-Name:[/faktory]] {} <nil> 0 [] true faktory-ui map[] map[] <nil> map[] 127.0.0.1:49432 /retries?page=2 <nil> <nil> <nil> 0xc000090280}
I 2021-05-05T21:09:11.994Z GET /retries?page=2 2.285647ms

Request No Proxy:
&{GET /retries?page=2 HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9] Connection:[keep-alive] Cookie:[csrf_token=+v6+UvZNkp4NHanHU/mByw5xQBTXAgSFlK5L0lu18fs=] Sec-Ch-Ua:[" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36]] {} <nil> 0 [] false localhost:7420 map[] map[] <nil> map[] 127.0.0.1:49436 /retries?page=2 <nil> <nil> <nil> 0xc00012ed80}
I 2021-05-05T21:09:19.510Z GET /retries?page=2 582.405µs

Looks like when it goes through the proxy the actual path it tries to hit is /retries%3Fpage=2?page=2 This doesn't match what faktory shows in the logs though. The log message right after shows as /retries?page=2

[lokiwins]

It looks like here is where it ends up breaking: https://github.com/contribsys/faktory/blob/a932536ce2450138e2658c56ecf79f8aeff0f0f7/webui/web.go#L166-L169

The request is correct prior to being adjusted.

Before Proxy Rewrite:
&{GET /faktory/retries?page=2 HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9] Connection:[close] Cookie:[csrf_token=mIlWhjEHfh5C/IyXl2V8wsOtJCx5clacBMLO1UmvNK8=] Sec-Ch-Ua:[" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36] X-Script-Name:[/faktory]] {} <nil> 0 [] true faktory-ui map[] map[] <nil> map[] 127.0.0.1:49804 /faktory/retries?page=2 <nil> <nil> <nil> 0xc0000a05c0}

After Proxy Rewrite:
&{GET /retries%3Fpage=2?page=2 HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9] Connection:[close] Cookie:[csrf_token=mIlWhjEHfh5C/IyXl2V8wsOtJCx5clacBMLO1UmvNK8=] Sec-Ch-Ua:[" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36] X-Script-Name:[/faktory]] {} <nil> 0 [] true faktory-ui map[] map[] <nil> map[] 127.0.0.1:49804 /retries?page=2 <nil> <nil> <nil> 0xc0000a05c0}

I 2021-05-05T21:58:51.028Z GET /retries?page=2 2.728579ms