Traefik Plugins: unable to create plugins due to read-only file system

[spatterIight]

Configuration

devture_traefik_configuration_extension_yaml: |
  experimental:
    plugins:
      geoblock:
        moduleName: "github.com/PascalMinder/geoblock"
        version: "v0.2.8"

Error:

Sep 04 01:55:12 lainon traefik[13393]: time="2024-09-04T01:55:12Z" level=error msg="Plugins are disabled because an error has occurred." error="unable to create plugins client: unable to create directory /plugins-storage/sources: mkdir plugins-storage: read-only file system"

Work-around

I was able to temporarily work-around this issue by using localPlugins and mounting the plugin

devture_traefik_configuration_extension_yaml: |
  experimental:
    localPlugins:
      geoblock:
        moduleName: github.com/PascalMinder/geoblock

devture_traefik_container_extra_arguments_custom:
  - "--volume /srv/traefik/geoblock:/plugins-local/src/github.com/PascalMinder/geoblock"

---

Going to look at making a pull request to handle this without needing the work-around

[spantaleev]

Sounds like we need an Ansible variable for easily removing the --read-only option applied to this container in templates/devture-traefik.service.j2.

Could you verify if removing this read-only flag is enough though? The filesystem will be writable (if removed), but Traefik may require some other capabilities for actually fully installing plugins.

[spatterIight]

Will do, I will investigate this and report back 🫡

[spatterIight]

It looks like even with the --read-only flag removed there is still an error:

Sep 07 22:28:55 lainon traefik[13555]: time="2024-09-07T22:28:55Z" level=error msg="Plugins are disabled because an error hasoccurred." error="unable to create plugins client: unable to create directory /plugins-storage/sources: mkdir plugins-storage: permission denied"

It looks like the user does not have permission to create the directory. Adding the following line to the service resolves this error: --tmpfs=/plugins-storage:rw,noexec,nosuid,size=8m

Plugins now work as expected without a local mount. Additionally, when using this the --read-only flag does not have to be removed -- so no change to the role is actually necessary.

So, this issue can probably be closed 🤔

[spantaleev]

I wonder if this directory path is configurable.

Mounting a tmpfs is a possibility, but.. it probably needs to be larger. And I'm not sure if it's great to use a tmpfs - restarting Traefik will mean it will need to download all its plugins again.

It's probably better if a persistent path is use, so that downtime is minimal and the risk of Traefik not starting (due to Github, etc., being down) is minimized. This can be a new directory (devture_traefik_plugins_dir_path) that is created by the role (tasks/install.yml) if some "plugins enabled" variable is true.

Maybe we won't even need to remove the --read-only argument if we mount a writable plugins directory where Traefik expects it (/plugins-storage). It's probably easiest if we use this default path and not customize it.

[spatterIight]

I think the size is ok, it seems like plugins are quite small: docker exec -it traefik /bin/sh -c 'du -sh /plugins-storage' -> 172.0K /plugins-storage

You are right that with tmpfs the plugins would need to be re-downloaded with each restart, and solving that to avoid problems if Github is down is pretty smart.

I will try to create a PR for this