Configuration Help

[FirbyKirby]

I've been trying to implement this plugin on my traefik network and I haven't been successful (host logs still show CF IPs rather then client IPs.) I was hoping someone could take a peek at my config files and tell me where I've gone wrong? Note that I see no errors or indications of the plugin in the log at any logging level.

One thing I'll note is that I am attempting to apply this middleware across all https traffic by defining it at the https entrypoint, and not at the service/router combo. Most of my hosts are other containers and I rely on the docker provider to handle those services, not service/router combos.

Static Config

NOTE: certificate config, logging config, and dashboard config removed for irrelevance.

global:
  checkNewVersion: true
  sendAnonymousUsage: true

serversTransport:
  insecureSkipVerify: true

entryPoints:
  # Not used in apps, but redirect everything from HTTP to HTTPS
  http:
    address: :80
    forwardedHeaders:
      trustedIPs: &trustedIps
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https

  # HTTPS endpoint, with domain wildcard
  https:
    address: :443
    forwardedHeaders:
      # Reuse list of Cloudflare Trusted IP's above for HTTPS requests
      trustedIPs: *trustedIps
    http:
      middlewares:
        - securityHeaders@file
        - cloudflarewarp@file

providers:
  providersThrottleDuration: 2s

  # File provider for connecting things that are outside of docker / defining middleware
  file:
    filename: /etc/traefik/fileConfig.yml
    watch: true

  # Docker provider for connecting all apps that are inside of the docker network
  docker:
    watch: true
    network: proxy    # Add Your Docker Network Name Here
    # Default host rule to containername.domain.example
    defaultRule: "Host(`{{ lower (trimPrefix `/` .Name )}}.HOSTNAME.TLD`)"    # Replace with your domain
    swarmModeRefreshSeconds: 15s
    exposedByDefault: false
    #endpoint: "tcp://dockersocket:2375" # Uncomment if you are using docker socket proxy

#Traefik Plugins
experimental:
  plugins:
    #Get the real IP of the client from Cloudflare and pass that to the host.
    cf-warp:
      moduleName: github.com/BetterCorp/cloudflarewarp
      version: v1.3.0

Dynamic Config

NOTE: Additional routers/services removed for redundancy, as well as some unused, but defined middleware.

http:

  ## EXTERNAL ROUTING - Only use if you want to proxy something manually ##
  routers:
    # Home Assistant routing
    homeassistant:
      entryPoints:
        - https
      rule: 'Host(`HOST.DOMAIN.TLD`)'
      service: homeassistant
      middlewares:
        #- "authentik"  #Uncomment when your ready to use authentik.

  ## SERVICES ##
  services:
    # Home Assistant service
    homeassistant:
      loadBalancer:
        servers:
          - url: http://192.168.1.4:8123/

  ## MIDDLEWARES ##
  middlewares:
    # Only Allow Local networks
    local-ipwhitelist:
      ipWhiteList:
        sourceRange:
          - 127.0.0.1/32 # localhost
          - 192.168.1.1/24 # LAN Subnet
          - 10.0.0.1/24 # LAN subnet

    #Authentik auth guard
    authentik:
      forwardauth:
        address: http://authentik:9000/outpost.goauthentik.io/auth/traefik
        trustForwardHeader: true
        authResponseHeaders:
          - X-authentik-username
          - X-authentik-groups
          - X-authentik-email
          - X-authentik-name
          - X-authentik-uid
          - X-authentik-jwt
          - X-authentik-meta-jwks
          - X-authentik-meta-outpost
          - X-authentik-meta-provider
          - X-authentik-meta-app
          - X-authentik-meta-version

    # Security headers
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          X-Forwarded-Proto: "https"
          server: ""
        sslProxyHeaders:
          X-Forwarded-Proto: https
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true

    gzip:
      compress: {}

    cloudflarewarp:
      plugin:
        cf-warp:
          disableDefault: false

# Only use secure ciphers - https://ssl-config.mozilla.org/#server=traefik&version=2.6.0&config=intermediate&guideline=5.6              
tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

Any suggestions on getting this integrated globally would be greatly beneficial. If I can't get it defined globally, I'd love to get it implemented as middleware for all the dockers traefik is proxying via the docker provider.

[mrinc]

So, it looks doable globally on an entrypoint.... but ...

    http:  
      middlewares:  
        - securityHeaders@file  
        - cloudflarewarp@file  

#Traefik Plugins  
experimental:  
  plugins:  
    #Get the real IP of the client from Cloudflare and pass that to the host.  
    cf-warp:  
      moduleName: github.com/BetterCorp/cloudflarewarp  
      version: v1.3.0  

Your plugin name defined here is cf-warp not cloudflarewarp as you are trying to reference it.

What I recommend doing is change the plugins name to cloudflarewarp and then see if the logs on traefik boot say anything.

#Traefik Plugins  
experimental:  
  plugins:  
    #Get the real IP of the client from Cloudflare and pass that to the host.  
    cloudflarewarp:  
      moduleName: github.com/BetterCorp/cloudflarewarp  
      version: v1.3.0  

ref:
https://doc.traefik.io/traefik/routing/entrypoints/#middlewares
https://github.com/BetterCorp/cloudflarewarp/blob/master/test/traefik.toml

[FirbyKirby]

Great catch in the name mismatch. I made the changed (removed all reference to cf-warp and replaced with cloudflarewarp) in both the static and dynamic (file) config for traefik and restarted. here's a look at the log on reboot.

time="2023-05-12T22:51:02-05:00" level=info msg="Traefik version 2.10.1 built on 2023-04-27T14:52:35Z"
time="2023-05-12T22:51:02-05:00" level=info msg="Stats collection is enabled."
time="2023-05-12T22:51:02-05:00" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2023-05-12T22:51:02-05:00" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2023-05-12T22:51:02-05:00" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider *file.Provider"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider *traefik.Provider"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider *docker.Provider"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-05-12T22:51:03-05:00" level=info msg="Starting provider *acme.Provider"
time="2023-05-12T22:51:03-05:00" level=info msg="Testing certificate renew..." providerName=letsencrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"

I can turn on debug logging if it will help, but I'm not seeing any errors. In the client I'm testing with, a remote connection is still logged in the http.log file as a cloudflare IP.

Here's a review of the last three log entries in my clients http.log file.

172.71.166.17 - F1rby [12/May/2023:22:58:43 -0500] "GET /ui/getFile/logs/L2NvbmZpZy9odHRwLmxvZw/100/1000?sort=tails HTTP/1.1" 200 23979 "https://notifiarr.HOSTNAME.TLD/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0" - 8773ms
 172.71.166.17 - F1rby [12/May/2023:22:58:53 -0500] "GET / HTTP/1.1" 200 457867 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0" - 86ms
172.71.166.17 - F1rby [12/May/2023:22:58:56 -0500] "GET /ui/getFile/logs/L2NvbmZpZy9hcHAubG9n/100/0?sort=tails HTTP/1.1" 200 11795 "https://notifiarr.HOSTNAME.TLD/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0" - 437ms

(Removed my domain info.)

Any other thoughts? Happy to share more logs, and I appreciate your feedback and suggestions.

[mrinc]

We don't replace the caller IP, so possible the ip ref in the logs doesn't look at the headers.
Try the whoami docker container, and see what info you get back?

[FirbyKirby]

I think you nailed it. I belive the client isn't referencing the X-Real-IP or the X-Forwarded-For header. After spinning up the whoami container, I spent some time getting it setup through traefik and Authentik (my authentication middleware) to replicate the exact setup this client is using. Here's what I get from whoami when I call that container externally.

Hostname: HOSTNAME
IP: 127.0.0.1
IP: 172.18.0.4
RemoteAddr: MY PROXY IP:53032
GET / HTTP/1.1
Host: whoami.HOSTNAME.TLD
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.5
Cdn-Loop: cloudflare
Cf-Connecting-Ip: MY REAL IP
Cf-Ipcountry: US
Cf-Ray: 7c6ffd35a9e0e772-DFW
Cf-Visitor: {"scheme":"https"}
Cookie: authentik_proxy_s8jvt8eR=AUTH TOKEN
Dnt: 1
Priority: u=1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
X-Authentik-Email: MY EMAIL
X-Authentik-Groups: MY USERS GROUPS
X-Authentik-Jwt: ANOTHER TOKEN
X-Authentik-Meta-App: whoami
X-Authentik-Meta-Jwks: http://localhost:8000/application/o/whoami/jwks/
X-Authentik-Meta-Outpost: authentik Embedded Outpost
X-Authentik-Meta-Provider: Who Am I? Forward Auth
X-Authentik-Meta-Version: goauthentik.io/outpost/2023.4.1
X-Authentik-Name: MY NAME
X-Authentik-Uid: 83b0660e1ee691da74f43588c586a2f8ccc175c47c3b6a68adec8c1b73a1bfc4
X-Authentik-Username: MY USERNAME
X-Forwarded-For: MY REAL IP, 172.71.167.131
X-Forwarded-Host: whoami.wondermutt.net
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: aaaafa624daf
X-Is-Trusted: yes
X-Real-Ip: MY REAL IP

Again, I santized that response as much as I could. All the "MY REAL IP" entries were matching IPs that I confirmed to be my real external IP.

So, I think this is resolved. Seems to be working precisely as it should be. My client just isn't using the right header(s).

[FirbyKirby]

That whoami container is suprisingly useful by the way. Great troubleshooting step!

[mrinc]

The whoami container is really simple and handy to work out what is cooking when you've got a custom setup with things in the middle etc... I find it much nicer than using helloworld for the default container for tutorials.

Glad I could help :)