search

BetterCorp / cloudflarewarp

MIT License
85 stars 6 forks source link

unexpected comma in X-Forward-For and empty X-Real-IP when local #19

Closed jonwilliams84 closed 1 year ago

jonwilliams84 commented 1 year ago

I have two tiers of traefik - dont ask why, long story, but tier one is in swarm (serving my public domain) and tier two is in k8s (mostly for internal stuff unless I chose to expose publicly via file provider on tier one).

I have been struggling to get the real client IP in my pod access logs, I always get the IP of my tier one traefik, even with forwardHeaders and trustedIPs configured on my tier 2 with the ip of my tier 1.

I have added you plugin to my tier 2 traefik and bingo, success - hitting my pods gives me the actual IP address of my client in my logs as long as is configure the plugin with trustip: 10.32.100.5/32 (tier 1 proxy) Result!

However, if I hit a pod behind tier 2 from inside of my network (dns record pointing at tier 1) I get the following:

X-Forwarded-For: , 10.32.100.5
X-Is-Trusted: yes
X-Real-Ip:

Notice the comma and missing X-Real-IP.

Any ideas how I can get around this?

mrinc commented 1 year ago

Hey @jonwilliams84

Please could you post the config used?

Although you aren't using the plugin for the intended use case, glad it's working for other use cases :P

I think this here is the line related to the issue: https://github.com/BetterCorp/cloudflarewarp/blob/master/cloudflarewarp.go#L115

It's marked as trusted since the IP is in your trust list, but because this plugin was specific for CloudFlare, it looks for the CF-Connecting-IP header, which most likely isn't there if you're connecting directly to it.

So what I think needs to be done is your tier 1 server IP is to be defined only and disableDefaults. 

disableDefault: false
trustip:
  - "{tier 1 server IP}/32"

What this would do is allow the forming of requests from your tier 1 to work while ignoring requests from any direct connections on your tier 2.

Would that work for your use case?

jonwilliams84 commented 1 year ago

Hey @jonwilliams84

Please could you post the config used?

Although you aren't using the plugin for the intended use case, glad it's working for other use cases :P

I think this here is the line related to the issue: https://github.com/BetterCorp/cloudflarewarp/blob/master/cloudflarewarp.go#L115

It's marked as trusted since the IP is in your trust list, but because this plugin was specific for CloudFlare, it looks for the CF-Connecting-IP header, which most likely isn't there if you're connecting directly to it.

So what I think needs to be done is your tier 1 server IP is to be defined only and disableDefaults.

disableDefault: false
trustip:
  - "{tier 1 server IP}/32"

What this would do is allow the forming of requests from your tier 1 to work while ignoring requests from any direct connections on your tier 2.

Would that work for your use case?

Hi, Thanks for the reply...I have tried the config you suggest - but still get the additional comma :-(

I have also tried moving the plugin to my tier 1 traefik (removing it from the tier 2), but as expected - this works fine for both local and external traffic, but the true-client-ip is not preserved when it hits my pods and I just see the IP of my tier 1 in my access logs again.

Instead of overwriting the X-Forwarded-Ip and X-Real-Ip headers could there be some logic that adds works out if its from a local subnet (or even all RFC1918 networks) and then appends to the X-Forwarded-For and leaves the X-Real-IP? Or if the traffic is coming from a trustedIP and the Cf-Connecting-IP doesn't exist it uses the original headers?

jonathann92 commented 1 year ago

@jonwilliams84

I have cloudflared and traefik running inside a docker container in the same docker network. The ip address of my cloudflared container is 172.22.0.X.

Working Result

I set the trustip to 172.16.0.0/12 which includes the IP addresses in the docker network. Optionally, I could have set it to 172.22.0.0/24 if I wanted to narrow it down to just the docker network's IP range.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 172.16.0.0/12  # docker network ip cidr

Response from whoami with DNS pointing to traefik via internal IP

X-Forwarded-For: 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 192.168.1.X

Response from whoami via cloudflare

Note:

X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip: XX.XXX.XXX.XXX

NOT Working Result

I set the trustip to 192.168.0.0/16 which includes the IP addresses for the computers in my local network.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 192.168.0.0/16  # possible computer ip range

Response from whoami

X-Forwarded-For: , 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip:

Response from whoami via cloudflare

Note:

X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 172.22.0.X
github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

mrinc commented 1 year ago

Hey @jonwilliams84 Please could you post the config used? Although you aren't using the plugin for the intended use case, glad it's working for other use cases :P I think this here is the line related to the issue: https://github.com/BetterCorp/cloudflarewarp/blob/master/cloudflarewarp.go#L115 It's marked as trusted since the IP is in your trust list, but because this plugin was specific for CloudFlare, it looks for the CF-Connecting-IP header, which most likely isn't there if you're connecting directly to it. So what I think needs to be done is your tier 1 server IP is to be defined only and disableDefaults.

disableDefault: false
trustip:
  - "{tier 1 server IP}/32"

What this would do is allow the forming of requests from your tier 1 to work while ignoring requests from any direct connections on your tier 2. Would that work for your use case?

Hi, Thanks for the reply...I have tried the config you suggest - but still get the additional comma :-(

I have also tried moving the plugin to my tier 1 traefik (removing it from the tier 2), but as expected - this works fine for both local and external traffic, but the true-client-ip is not preserved when it hits my pods and I just see the IP of my tier 1 in my access logs again.

Instead of overwriting the X-Forwarded-Ip and X-Real-Ip headers could there be some logic that adds works out if its from a local subnet (or even all RFC1918 networks) and then appends to the X-Forwarded-For and leaves the X-Real-IP? Or if the traffic is coming from a trustedIP and the Cf-Connecting-IP doesn't exist it uses the original headers?

Hey,

The plugin is purely for CloudFlare/CloudFlare WARP/CloudFlare Tunels. So, adding the code to be able to work as a basic reverse proxy IP checker isn't in it's scope.
From what I know there are other plugins that work as basic IP validators though.

The main thing here is we validate based on the CF- connecting IP rather than x-forwarded for.

mrinc commented 1 year ago

@jonwilliams84

I have cloudflared and traefik running inside a docker container in the same docker network. The ip address of my cloudflared container is 172.22.0.X.

Working Result

I set the trustip to 172.16.0.0/12 which includes the IP addresses in the docker network. Optionally, I could have set it to 172.22.0.0/24 if I wanted to narrow it down to just the docker network's IP range.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 172.16.0.0/12  # docker network ip cidr

Response from whoami with DNS pointing to traefik via internal IP

X-Forwarded-For: 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 192.168.1.X

Response from whoami via cloudflare

Note:

  • XX.XXX.XXX.XXX is my public IP address
  • 172.22.0.X is my cloudflare container's IP address
X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip: XX.XXX.XXX.XXX

NOT Working Result

I set the trustip to 192.168.0.0/16 which includes the IP addresses for the computers in my local network.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 192.168.0.0/16  # possible computer ip range

Response from whoami

X-Forwarded-For: , 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip:

Response from whoami via cloudflare

Note:

  • XX.XXX.XXX.XXX is my public IP address
  • 172.22.0.X is my cloudflare container's IP address
X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 172.22.0.X

Hey,

The plugin seems to doing it's job.
Like noted above, it's purely for CF- ips, not a forwarded for proxy.

So what happens in your examples, is that when you connect with your not working examples:
Response from whoami: It's trying to pull the CF- connecting ip which doesn't exist since it's local thus returning blank.
Response from whoami via cloudflare - Well the IP range of your CF server isn't listed anymore, so it will class all connections from your cf server as unsafe and not proxy ip them.

PrplHaz4 commented 1 year ago

@jonwilliams84 I have cloudflared and traefik running inside a docker container in the same docker network. The ip address of my cloudflared container is 172.22.0.X.

Working Result

I set the trustip to 172.16.0.0/12 which includes the IP addresses in the docker network. Optionally, I could have set it to 172.22.0.0/24 if I wanted to narrow it down to just the docker network's IP range.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 172.16.0.0/12  # docker network ip cidr

Response from whoami with DNS pointing to traefik via internal IP

X-Forwarded-For: 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 192.168.1.X

Response from whoami via cloudflare

Note:

  • XX.XXX.XXX.XXX is my public IP address
  • 172.22.0.X is my cloudflare container's IP address
X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip: XX.XXX.XXX.XXX

NOT Working Result

I set the trustip to 192.168.0.0/16 which includes the IP addresses for the computers in my local network.

Traefik dynamic file config

        cloudflarewarp:
          disableDefault: false
          trustip: # Trust IPS not required if disableDefault is false - we will allocate Cloud Flare IPs automatically
            - 192.168.0.0/16  # possible computer ip range

Response from whoami

X-Forwarded-For: , 192.168.1.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: yes
X-Real-Ip:

Response from whoami via cloudflare

Note:

  • XX.XXX.XXX.XXX is my public IP address
  • 172.22.0.X is my cloudflare container's IP address
X-Forwarded-For: XX.XXX.XXX.XXX, 172.22.0.X
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: Redacted
X-Is-Trusted: no
X-Real-Ip: 172.22.0.X

Hey,

The plugin seems to doing it's job. Like noted above, it's purely for CF- ips, not a forwarded for proxy.

So what happens in your examples, is that when you connect with your not working examples: Response from whoami: It's trying to pull the CF- connecting ip which doesn't exist since it's local thus returning blank. Response from whoami via cloudflare - Well the IP range of your CF server isn't listed anymore, so it will class all connections from your cf server as unsafe and not proxy ip them.

Shouldn't it not attempt to add to XFF when CF-Connecting-IP is blank or doesn't exist (eliminating the errant comma)?

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days