st2web fails in rootless environment

[joschi36]

To increase security in companies clusters its common practice to run root-less containers. This can be enforced in Kubernetes with PodSecurityPolicies. For instance Openshift clusters have this restrictive policies on by default.

The st2web container however is currently using the library/nginx image which doesn't support to be run as non-root. However, the bitnami/nginx image is doing exactly this to improve security.

I would suggest adopting to bitnami images, as they are often better suited to Kubernetes and since you are already using the bitnami ecosystem.

References:

• https://github.com/StackStorm/st2web/blob/master/Dockerfile-nginx
• https://github.com/StackStorm/st2web/blob/master/Dockerfile-nginx-dev

I've first tried to create a Pull Request over at the st2web repo but had some issues creating the image locally. Also, this is my first try with installing stackstorm in general and also with contributing, so I thought maybe I should create an issue first.

Replacing the images thought shouldn't be very hard as only few directories need to be changed.

Why use a non-root container?

Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers in our docs.

from: bitnami/nginx README.md

[arm4b]

Thanks for the report, that's helpful :+1: The st2web container is based on ubuntu:bionic and we build it ourselves by installing nginx. Check the source code here: https://github.com/StackStorm/st2-dockerfiles/blob/master/st2web/Dockerfile

Are there any security enhancements you could identify and propose for that specific Dockerfile?

[joschi36]

@armab Thanks for your quick response! :)

1. You could use instead of ubuntu the bitnami/nginx image which is based on minideb (Minified Debian from Bitnami) which already uses some of the best practice of container images.
2. Or we could install nginx not in user root, but I think this is more complicated than just using what bitnami has done.

~If you want to follow the first path I could try and submit a PR.~ I just saw that also the second path is not that complicated. https://stackoverflow.com/a/42329561 Maybe I will look at that first.

[joschi36]

I'm currently working and trying both options. But what I don't get is why you don't have the Dockerfile in the st2web directory. I think it would make container creation more robust, secure and minimized. Maybe you @armab can clarify this to me?

[arm4b]

Thanks, @joschi36, that would be indeed very helpful :+1: I'd prefer the 2nd approach trying to modify the existing Dockerfile running nginx as a USER, if it doesn't get really complicated.

Talking about the Dockerfiles, at this moment we keep them all in one place with the unified build, deploy and release pipeline https://github.com/StackStorm/st2-dockerfiles/ instead of scattering them across the different repositories. Both have pros/cons and I guess there were also historical reasons why it worked that way.

[cwilson21]

RE: Bitnami Approach

@armab @joschi36 while in most cases I would agree with you for this approach this one I cannot agree with you. This is throwing yet another dependency that Bitnami does not build with multiple architectures. Currently bitnami/nginx like bitname/rabbitmq is only built for the AMD64 arch type. One of the great things we find with the k8s version of stackstorm is it is easier to develop in because it is easy to spin up a local stackstorm stack with minikube/docker-desktop to do testing in. However with the introduction of daily driving ARM based machines, like Macbook Pros not having images in all Architectures causes issues.

[Kishore1705]

what is the option that we can use here? My st2web is failing because of non-root , I'm trying to deploy OpenShift and OpenShift by default doesn't give root access to the containers

[ZoeLeah]

There is a pull request (or was) to make st2web rootless. It's only about the ports for nginx. Above 1024 you don't need root access. I will search for the pr.

[ZoeLeah]

https://github.com/StackStorm/st2-dockerfiles/pull/66

[Kishore1705]

StackStorm/st2-dockerfiles#66

so i should build my own Dockerfile? and then is there any change that has to be done on Helm chart after this?

[ZoeLeah]

We were building our own container images and updated the helm chart. But I don't know how to manipulate the helm chart to use only the web container from a private registry.

[Kishore1705]

why can't we modify directly on the Deployment after installing the helm chart?

[ZoeLeah]

Never tried it

[Kishore1705]

there is error when trying to run st2web /bin/bash: /etc/nginx/conf.d/st2.conf: Permission denied getting this error on st2web

there is no file as such when i see the path

[Kishore1705]

sh: 1: cannot create /root/.st2/config: Permission denied , getting this error when i try to run Jobs/st2actionrunner for st2client , tried to modify the Dockerfile chmod 777 /root/.st2/config but it says file not found exception during docker build

[jk464]

My PR referenced above (https://github.com/StackStorm/st2-dockerfiles/pull/66) also changes permissions on some files to allow them to be read/written by a non-root user (I did hard code it to the nginx user)

I've also got a PR for the helm chart here (https://github.com/StackStorm/stackstorm-k8s/pull/400) which makes the required changes to run st2web (and all the other st2 services) as rootless containers w/o any privilege escalation

[Kishore1705]

can i take this branch and install directly, will that work without all these issues? i'm facing these permission issues on every step.

[jk464]

You'll want to build my branch of the docker images locally push them to your internal registry. (the st2-dockerfiles contains a make file for building the images)

And then you should also be able to helm install my branch of the helm chart aswell.

Also in the values.yaml of the helm chart you'll want to point towards your internal registry by having:

##
## Docker image settings, applied to all StackStorm pods
##
image:
  # This should point to the folder that contains `st2web` docker images etc
  repository: "<internal_registry_url>/stackstorm"
  # Whatever tag you tagged the docker images with
  tag: "<tag>"

That's hopefully everything you need to get going - Although I will note I don't believe anyone has used either of my branches outside the environment I've deployment them too - so there's a chance you'll hit an issue with an assumption I have made - if so please let me know.

[Kishore1705]

i guess only the st2web image has been changed , apart from that everything else remains the same?

[jk464]

Yes the other st2 images don't require any changes to run rootless

[Kishore1705]

When i try to login to st2 client it says permission denied because st2 folder has root level permission, can you please check once?

[Kishore1705]

ERROR: Failed to log in as st2admin: [Errno 13] Permission denied: '/.st2', this is the error which i'm getting