X-Forwarded-For or X-Real-IP is not forwarding client ip to backend

[mahdiadnan]

After setting the mentioned header in the webserver 'nginx', the backend still receives DNS queries from nginx IP and not from the client IP. Even in the logs, I only see the IP of the web server and not the client. Is it bugged or I'm missing something here?

from nginx config: ... proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; ... proxy_pass http://ungound-dns;

from DOH config: ... log_guessed_client_ip = true

[m13253]

I haven't seen this problem before.

Would you please try to do a packet sniff between Nginx and doh-server using either Wireshark or tcpdump?

[m13253]

By the way, the IP guessing feature ignores any private IP in the proxy chain and detects only the first public IP in the proxy chain. This is designed for GeoDNS -- because a private IP can not indicate any geographical information.

[mahdiadnan]

In the debug logs of nginx I can see that nginx has set the headers correctly. and from the pcap file I can the headers are set with the client public IP.

[m13253]

Thank you for your info. I will try to diagnose this problem.

[m13253]

By the way, for each request there will be two lines of log:

100.100.100.100 - - [29/Feb/2020:08:46:00 +0000] "example.com. IN A"
127.0.0.1 - - [29/Feb/2020:08:46:00 +0000] "POST /dns-query HTTP/1.1" 200 121 "" "Go-http-client/1.1"

One of the 2 lines are generated by my code, which contains the guessed client IP. The other line is generated by the Gorilla web toolkit. And I just currently have no idea how to change the IP address in this line.

[qyb]

I'm the log_guessed_client_ip patch author. I suggest set whatever environment/argument to remove the gorilla log and add dns-over-https's user-agent log

[jordanbtucker]

I'm having the same issue with Apache. I have verified that Apache is sending the X-Forwarded-* headers, but the doh-server is logging 127.0.0.1 for both log lines. I've put relevant config and log files in this gist.

https://gist.github.com/jordanbtucker/0988c34e54089ca11d850840986bf56e

[m13253]

Thank you for your information. I think I need to replace the code provided by Gorilla and rewrite it ourselves.

[buckaroogeek]

First off - I am not proficient in Go so I could be way off base.

Could the ProxyHeaders handler in the gorilla toolkit be used to generate the needed data? As far as I can tell the only handler from the gorilla handlers package in use is the CombinedLoggingHandler. There are some notes in the documentation that using ProxyHeaders when the go http server is not behind a proxy creates a potential vulnerability.

Can the log_guessed_client_ip setting in doh_server.conf be used when set to true to (1) assume a proxy is used upstream and (2) add the ProxyHeader handler to the list of handlers that are used?

I might also note, the in the use case mentioned by @jordanbtucker, so requests coming from 10.0.1.2. If I read the findClientIP function correctly, XForwardedFor and XRealIP only return an IP if a global IP is the source. Am I understanding that accurately?

[jordanbtucker]

@buckaroogeek Looks like you are correct about the global IP issue. It checks for a global IP here, here, and here. Perhaps we could have an option to return any forwarded IP rather than just global ones?

[aniqueta]

I've had the same issue since version 2.0.1. Running 2.2.1 now. Nginx is setup to pass the IP address to DOH Server. However, the IP does not get passed to the backend, in my case, Pihole-FTL. Until this thread, I thought this was just normal behavior. With Pihole there's value in seeing the IP to understand which devices are making the most blocked requests. Thanks.

[m13253]

Thank you for your report!

I am currently rewriting the part from Gorilla and soon I will be able to get this thing fixed.

[m13253]

Please check the latest version and see whether it fixes this problem.

Note that X-Forwarded-For or guessed client IP is not used in HTTP log due to security concerns. But you should be able to pass X-Real-IP from the frontend nginx/Apache/Caddy to doh-server now.

[hornetmadness]

Please check the latest version and see whether it fixes this problem.

Note that X-Forwarded-For or guessed client IP is not used in HTTP log due to security concerns. But you should be able to pass X-Real-IP from the frontend nginx/Apache/Caddy to doh-server now.

Working for me now.

[aniqueta]

@m13253 Thanks for the fixes. The 2.2.2 update seems to have partially fixed the issue, at least for my configuration. doh-server is logging the right external IP (verbose = true and log_guessed_client_ip = true). However, the upstream provider to doh-server (in my case pihole-FTL) does not see the external IP. It still just sees 127.0.0.1. Is that normal? If so, is there a way to config doh-server pass the IPs to the upstream provider? Thanks.

[jordanbtucker]

I am having the same issue as @aniqueta. This issue is not closed for me.

[aniqueta]

@m13253 Should we create a new issue for this, since this one is closed, or would it be possible to reopen this? Any additional info we can help with? Thank you!

[m13253]

Reopened the issue. But currently I don't have an idea of how to fix it. Maybe we can work out together.

[jordanbtucker]

I think the problem is that findClientIP is only returning IP addresses that pass jsondns.IsGlobalIP. I want to know which IP address on my local network made a query.

https://github.com/m13253/dns-over-https/blob/4f46b89febf08ad906be51765b1703cb61e2f90a/doh-server/server.go#L248-L280

[jordanbtucker]

After a second look, it appears my issue is different from @aniqueta's.

~To answer @aniqueta's question, I do not think it is possible for DoH to forward the original IP address upstream (pihole in your case, BIND in mine). Since DNS servers receive requests over UDP and/or TCP, there is no way to include the original IP address in those packets, and as far as I know, the DNS protocol does not provide a way to include that information either.~

[m13253]

as far as I know, the DNS protocol does not provide a way to include that information either.

Edns0-Client-Subnet protocol

[m13253]

I want to know which IP address on my local network made a query.

I guess according to the code at https://github.com/m13253/dns-over-https/blob/4f46b89febf08ad906be51765b1703cb61e2f90a/doh-server/ietf.go#L99-L106 doh-server is already trying its best to log the client IP address.

I want doh-server to satisfy two requirements: 1) The IP address I want to forward to upstream using ECS must be global IP address, so the upstream DNS server can know which country the client is from and resolve to a fast edge server. Providing local IP address does not help with this. 2) Since a typical doh-server deployment usually includes multiple cache servers (both at HTTP side and DNS side), skipping local addresses are designed to skip these cache server's addresses.

I understand that you would like to know which host submitted the original DNS request, but I cannot figure out a good way to implement this and also fulfill the above two requirements. Therefore I am not sure how to solve the problem right now. Let's discuss about it.

[jordanbtucker]

Edns0-Client-Subnet protocol

Oh, that's right. Sorry, it's been a while since I worked on this issue with my DNS server.

[m13253]

Another reason my original design is to skip all local addresses are because the Google DoH protocol (at that time the IETF thing did not exist) is human readable. Therefore, an Apache log is enough to log the client's address, doh-server's log is only used to debug ECS deployment.

But nowadays the IETF DoH uses a base64-encoded request, which makes the Apache log hard to decipher (still able, though). So that's how this issue comes into place.

[jordanbtucker]

That makes sense. In my use case, doh-server will never see an X-Forwarded-For or X-Real-IP with a global IP address because it only serves my local network. It's not public facing. So a configuration option to forward non-global IP address via ECS would be ideal for my use case. Then BIND could do all the logging, and I wouldn't need log_guessed_client_ip at all.

[m13253]

Then BIND could do all the logging

Maybe you can try to log the client's IP at BIND side, because that can solve the problem and, you know, since that is a valid workaround, I am a little bit lazy about this. Also, if anyone implements this function, feel free to send me a Pull Request.

At this time, I think we can temporarily stick to the UNIX Philosophy -- combine with a BIND or unbound or dnsmasq, since we will surely need a cache after all.

[jordanbtucker]

I would love for BIND to do all the logging, but BIND cannot get the original IP address from doh-server because it only includes global IP addresses in the ECS info. If there was an option for doh-server to include local IP addresses in the ECS info, then that would solve my problem.

[m13253]

If there was an option for doh-server to include local IP addresses in the ECS info

That would confuse the upstream resolver and probably give you a resolution results in a different country, or use the IP address of their best guess.

[m13253]

By the way, if you are satisfied with current solution (or workaround), please notify me so I can release version 2.2.3. Otherwise, I will delay the release and see if we can come up with better solutions.

[aniqueta]

I do think what @m13253 is trying to achieve matches my use case and addresses my reason to reopen the ticket, although the above conversation is difficult to track, so I'd like to confirm.

My DOH setup takes in queries from the web, and so I do want the upstream provider (Pi-hole, running on the same machine as DOH) to log the global IP address originating the DNS query. For security, there is a firewall and reverse proxy protecting DOH exposed to the web, and I do not care to log its IP. I think that's exactly the functionality @m13253 is trying to maintain, per this comment:

I want doh-server to satisfy two requirements:

1. The IP address I want to forward to upstream using ECS must be global IP address, so the upstream DNS server can know which country the client is from and resolve to a fast edge server. Providing local IP address does not help with this.

2. Since a typical doh-server deployment usually includes multiple cache servers (both at HTTP side and DNS side), skipping local addresses are designed to skip these cache server's addresses.

Will the proposed solution here work to log the global IP address on Pi-hole (a dnsmasq variant)? I'm not knowledgeable enough about ECS, and whether that will do the trick. Thank you!

[m13253]

Solved with #92

[aniqueta]

@m13253 Apologies for my very delayed reply on this. Just got around to testing this, and it solves my use case with pihole as well. Thanks and good to close this, I think.