Middleware cannot be downloaded in some cases

mathieuHa commented 4 days ago

Welcome!

[x] Yes, I've searched similar issues on GitHub and didn't find any.
[x] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I have some report from users stating they cannot download a middleware from plugins.traefik.io.

It has been reported initially in the repository of the plugin. maxlerebourg/crowdsec-bouncer-traefik-plugin/issues/197

I could not reproduced locally by multiples users stated they could not download the last version.

I believe it is an issue of Traefik, or Traefik Infrastructure and not related to the "plugin code".

I saw one other exemple from another middleware reported there https://github.com/traefik/traefik/issues/11296

What did you see instead?

Traefik plugin.io malfunctionning ?

What version of Traefik are you using?

3.2.1

What is your environment & configuration?

services:
  traefik:
    image: "traefik:v3.2.1"
    container_name: "traefik"
    restart: unless-stopped
    command:
      - "--accesslog"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"

      - "--experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.bouncer.version=v1.3.5"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      # - './ban.html:/ban.html:ro'
      # - './captcha.html:/captcha.html:ro'
      - "logs:/var/log/traefik"
    ports:
      - 8000:80
      - 8080:8080
    depends_on:
      - 'crowdsec'

  whoami1:
    image: traefik/whoami
    container_name: "simple-service-foo"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      # Definition of the router
      - "traefik.http.routers.router-foo.rule=Path(`/foo`)"
      - "traefik.http.routers.router-foo.entrypoints=web"
      - "traefik.http.routers.router-foo.middlewares=crowdsec@docker"
      # Definition of the service
      - "traefik.http.services.service-foo.loadbalancer.server.port=80"

  whoami2:
    image: traefik/whoami
    container_name: "simple-service-bar"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      # Definition of the router
      - "traefik.http.routers.router-bar.rule=Path(`/bar`)"
      - "traefik.http.routers.router-bar.entrypoints=web"
      - "traefik.http.routers.router-bar.middlewares=crowdsec@docker"
      # Definition of the service
      - "traefik.http.services.service-bar.loadbalancer.server.port=80"
      # Definitin of the middleware
      - "traefik.http.middlewares.crowdsec.plugin.bouncer.enabled=true"
      # crowdseclapikey is the key to authenticate to crowdsec
      - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=FIXME-LAPI-KEY-1="
      # enable AppSec real time check
      - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecappsecenabled=true"
      # forwardedheaderstrustedips should be the IP of the proxy that is in front of traefik (if any)
      - "traefik.http.middlewares.crowdsec.plugin.bouncer.forwardedheaderstrustedips=172.21.0.5"

  crowdsec:
    image: crowdsecurity/crowdsec:v1.6.1-2
    container_name: "crowdsec"
    restart: unless-stopped
    environment:
      COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
      CUSTOM_HOSTNAME: crowdsec
      # We need to register one api key per service we will use
      BOUNCER_KEY_TRAEFIK: FIXME-LAPI-KEY-1=
    volumes:
      - ./acquis.yaml:/etc/crowdsec/acquis.yaml:ro
      - logs:/var/log/traefik:ro
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
    labels:
      - "traefik.enable=false"

volumes:
  logs:
  crowdsec-db:
  crowdsec-config:

This exemple could not reproduce the error, i'll complete when I get more information

If applicable, please paste the log output in DEBUG level

2024-11-21T15:37:05Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:237 > Loading plugins... plugins=["crowdsec-bouncer-traefik-plugin"]
2024-11-21T15:37:05Z DBG github.com/traefik/traefik/v3/pkg/plugins/plugins.go:30 > Loading of plugin: crowdsec-bouncer-traefik-plugin: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin@v1.3.5
2024-11-21T15:37:05Z DBG github.com/hashicorp/go-retryablehttp@v0.7.7/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/download/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/v1.3.5
2024-11-21T15:37:05Z ERR github.com/traefik/traefik/v3/cmd/traefik/traefik.go:242 > Plugins are disabled because an error has occurred. error="unable to set up plugins environment: unable to download plugin github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin: error: 404: <!DOCTYPE html> (...) </html>" plugins=["crowdsec-bouncer-traefik-plugin"]

kevinpollet commented 3 days ago

Hello @mathieuHa and thanks for raising this,

We have looked at our infrastructure logs and it looks like this was a transient error, and we apologize for that. Can you confirm that everything is working as expected?

saltydk commented 3 days ago

I've seen 3 different reports of this on our project discord on 3 different days for what it is worth, so this would suggest it isn't just a single blip.

saltydk commented 1 day ago

2 additional cases today.

JoshBeer commented 1 day ago

I can confirm this behaviour

ERR Plugins are disabled because an error has occurred. error="unable to set up plugins environment: unable to download plugin github.com/BetterCorp/cloudflarewarp: error: 404:

AkkelDeFakkel commented 1 day ago

Having the same issue. I’m using the crowdsec and cloudflarewarp plugin and they can’t be downloaded. Had to disable them in order for traefik to work. This should be a number 1 priority.

RobHofmann commented 1 day ago

Same issue here.

experimental:
  plugins:
    cloudflarewarp:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.3.3"
    crowdsec-bouncer-traefik-plugin:
      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      version: "v1.3.4"

    dummy.nl:
      rule: "Host(`www.dummy.nl`) || Host(`dummy.nl`)"
      tls:
        certResolver: letsencrypt
      service: dummy.nl@file
      entryPoints:
      - https
      middlewares:
      - referrerPolicy
      - compress
      - ipWhitelisting
      - cfReverseproxyRealIp
      - crowdsec-bouncer

    cfReverseproxyRealIp:
      plugin:
        cloudflarewarp:
          disableDefault: false
    crowdsec-bouncer:
      plugin:
        crowdsec-bouncer-traefik-plugin:
          enabled: true
          loglevel: WARN
          crowdsecLapiKey: <redacted>
          crowdsecMode: stream
          crowdsecAppsecEnabled: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecFailureBlock: true
          redisCacheEnabled: true
          redisCacheHost: "crowdsec-redis:6379"
          banHTMLFilePath: /pages/ban.html
          clientTrustedIPs: 
            - 192.168.0.0/19
          forwardedHeadersTrustedIPs: 
            - 173.245.48.0/20
            - 103.21.244.0/22
            - 103.22.200.0/22
            - 103.31.4.0/22
            - 141.101.64.0/18
            - 108.162.192.0/18
            - 190.93.240.0/20
            - 188.114.96.0/20
            - 197.234.240.0/22
            - 198.41.128.0/17
            - 162.158.0.0/15
            - 104.16.0.0/13
            - 104.24.0.0/14
            - 172.64.0.0/13
            - 131.0.72.0/22
            - 2400:cb00::/32
            - 2606:4700::/32
            - 2803:f800::/32
            - 2405:b500::/32
            - 2405:8100::/32
            - 2a06:98c0::/29
            - 2c0f:f248::/32

    dummy.nl:
      loadBalancer:
        passHostHeader: true
        servers:
          - url: http://dummy_nl

mloiseleur commented 37 minutes ago

Following user reports, we have taken the time to conduct deep investigation on how it can happens.

We have found a case where transient error (404) can be cached. => We have modified the infrastructure to avoid this case. We also purged the cache.

We'll continue to watch carefully the plugin infrastructure in the next days.

traefik / traefik