Cannot set readOnlyRootFilesystem for ui container

zsolooo commented 7 months ago

Describe the bug As a mandatory K8S cluster hardening policy in our organisation it's required to set readOnlyRootFilesystem for each container inside a pod. The actual issues are with the docker-entrypoint script: https://github.com/opencost/opencost/blob/develop/ui/docker-entrypoint.sh#L21 It tries to write a few files to the root volume (/var/www and /etc/nginx/conf.d). In case of /var/www it was possible to workaround the issue by attaching a writable emptyDir volume to /var/www, however the same approach was not possible with /etc/nginx/conf.d, because mounting a volume on that path hides files that are already existing on that path (entrypoint script tries to read config template to use for envsubst).

This issue makes the pod unable to start (CrashLoopBackoff).

To Reproduce with the tried workaround Steps to reproduce the behavior:

Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , chart-version: 1.32.0 Using the following helm values:

opencost:
 exporter:
   startupProbe:
     enabled: false
   livenessProbe:
     enabled: false
   readinessProbe:
     enabled: false
 ui:
   securityContext:
     capabilities:
       drop:
       - ALL
     allowPrivilegeEscalation: false
     readOnlyRootFilesystem: true
     runAsNonRoot: true
     runAsUser: 1000

Check the ui container logs with kubectl

See error:

cp: can't create '/var/www/MaterialIcons-Regular.12b3b105.woff': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.333251c4.ttf': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.b99eb5ce.woff2': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.e9e55c63.eot': Read-only file system
cp: can't create '/var/www/favicon.7eff484d.ico': Read-only file system
cp: can't create '/var/www/index.3406705b.css': Read-only file system
cp: can't create '/var/www/index.3406705b.css.map': Read-only file system
cp: can't create '/var/www/index.4170b679.js': Read-only file system
cp: can't create '/var/www/index.4170b679.js.map': Read-only file system
cp: can't create '/var/www/index.6d70e179.js': Read-only file system
cp: can't create '/var/www/index.6d70e179.js.map': Read-only file system
cp: can't create '/var/www/index.html': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js.map': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js.map': Read-only file system
cp: can't create '/var/www/logo.b9464e00.png': Read-only file system

To Reproduce with the described workaround (still failing) Steps to reproduce the behavior:

Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , version: 1.32.0 Using the following helm values:

opencost:
 exporter:
   startupProbe:
     enabled: false
   livenessProbe:
     enabled: false
   readinessProbe:
     enabled: false
 ui:
   securityContext:
     capabilities:
       drop:
       - ALL
     allowPrivilegeEscalation: false
     readOnlyRootFilesystem: true
     runAsNonRoot: true
     runAsUser: 1000
   extraVolumeMounts:
   - mountPath: /var/www
     name: ui-www
   - mountPath: /etc/nginx/conf.d
     name: ui-nginx-config
 extraVolumes:
 - name: ui-www
   emptyDir: {}
 - name: ui-nginx-config
   emptyDir: {}

Check the ui container logs with kubectl

See error:

'/opt/ui/dist/MaterialIcons-Regular.12b3b105.woff' -> '/var/www/MaterialIcons-Regular.12b3b105.woff'
'/opt/ui/dist/MaterialIcons-Regular.333251c4.ttf' -> '/var/www/MaterialIcons-Regular.333251c4.ttf'
'/opt/ui/dist/MaterialIcons-Regular.b99eb5ce.woff2' -> '/var/www/MaterialIcons-Regular.b99eb5ce.woff2'
'/opt/ui/dist/MaterialIcons-Regular.e9e55c63.eot' -> '/var/www/MaterialIcons-Regular.e9e55c63.eot'
'/opt/ui/dist/favicon.7eff484d.ico' -> '/var/www/favicon.7eff484d.ico'
'/opt/ui/dist/index.3406705b.css' -> '/var/www/index.3406705b.css'
'/opt/ui/dist/index.3406705b.css.map' -> '/var/www/index.3406705b.css.map'
'/opt/ui/dist/index.4170b679.js' -> '/var/www/index.4170b679.js'
'/opt/ui/dist/index.4170b679.js.map' -> '/var/www/index.4170b679.js.map'
'/opt/ui/dist/index.6d70e179.js' -> '/var/www/index.6d70e179.js'
'/opt/ui/dist/index.6d70e179.js.map' -> '/var/www/index.6d70e179.js.map'
'/opt/ui/dist/index.html' -> '/var/www/index.html'
'/opt/ui/dist/index.runtime.256cff63.js' -> '/var/www/index.runtime.256cff63.js'
'/opt/ui/dist/index.runtime.256cff63.js.map' -> '/var/www/index.runtime.256cff63.js.map'
'/opt/ui/dist/index.runtime.997a1df7.js' -> '/var/www/index.runtime.997a1df7.js'
'/opt/ui/dist/index.runtime.997a1df7.js.map' -> '/var/www/index.runtime.997a1df7.js.map'
'/opt/ui/dist/logo.b9464e00.png' -> '/var/www/logo.b9464e00.png'
running with BASE_URL=/model
/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Expected behavior Pod starts up correctly with readonly root filesystem

Screenshots Not relevant

Which version of OpenCost are you using? OpenCost: v1.109.0 Helm chart: v1.32.0

Additional context None

mkilchhofer commented 7 months ago

I also wanted to raise an issue for this. We have the same approach forcing every workload to set the field pod.spec.containers.securityContext.readOnlyRootFilesystem to true.

For /var/www there was a workaround as mentioned until

https://github.com/opencost/opencost/pull/2366/

was implemented. Now we are unable to workaround this.

Update: Found a recent PR which should fix this:

https://github.com/opencost/opencost/pull/2521

But it still seems not possible. @kaitimmer does it still work for you?

kaitimmer commented 7 months ago

But it still seems not possible. @kaitimmer does it still work for you?

Actually no, because I missed this second part. Currently, we run opencost with an exception to the rule. My hope is that this fixes all of the places where it is needed. But if you can check again, that would be great. 4-eyes more than 2 :)

jamalsms commented 3 months ago

I still get the message in version 1.111.0

/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Has anyone managed to overcome this problem?

morawat commented 2 months ago

has there been any fix or known workaround?

ervinb commented 1 month ago

This worked for me:

# values.yaml
opencost:
  ui:
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 1001

    extraVolumeMounts:
      - name: empty-var-www
        mountPath: /var/www

extraVolumes:
  - name: empty-var-www
    emptyDir: {}

opencost / opencost-ui

Cannot set readOnlyRootFilesystem for ui container #4