[BUG][OpenSearch] Only ReadOnly file system is allowed while installing OpenSearch via helm chart

[kunal-parsewar]

Describe the bug I am trying to install opensearch 1.2.3 helm chart in an AKS cluster but getting error as "Only Read-Only root file system container is allowed " and " Privilege escalation container is not allowed." Our security team is not ready to provide the exception for this helm chart deployment.

Have anyone faced this issue and how they resolved it without taking an exception from the security team.

Also , I would like to know why opensearch tries to modify the filesystem during installation ( my guess is for securityadmin.sh file ) and why opensearch image requires privilege escalation mode.

Security team wants to know the reason behind same and without those reasons they are not going to provide the exception.

To Reproduce Steps to reproduce the behavior:

1. Download 1.2.3 helm chart for opensearch.
2. Setup aks cluster.
3. Setup gatekeeper policies.
4. Try to install the helm chart.
5. You will see errors.

Expected behavior OpenSearch container should be installed without modifying filesystem and without any privilege escalation mode.

Plugins All default plugins are available.

OpenSearch : 1.2.3 AKS

[tlfeng]

@opensearch-project/engineering-effectiveness Could you provide suggestion for the issue of installation through Helm Charts?

[dblock]

Moving to the devops repo.

[gaiksaya]

Moving to helm-chart repo that looks more appropriate for this issue. Thanks!

[prudhvigodithi]

Hey @kunal-parsewar not sure if you have used the right helm chart. The latest chart version for OpenSearch is 2.6.1 and for OpenSearch Dashboard is 2.5.2 Chart repo link: OpenSearch OpenSearch-Dashboards Artifacthub Links: OpenSearch OpenSearch-Dashboards

Quick install commands:

helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo update
helm install opensearch opensearch/opensearch

[kunal-parsewar]

Hi @prudhvigodithi thanks for the reply , I think I am using pretty old chart ( version: 1.5.1 ).

Let me try installing the newer version with some changes according to our environment.

May i know if current chart version will not produce above raised errors?

[kunal-parsewar]

Just to clarify 1.2.3 is the OpenSearch version.

[prudhvigodithi]

Hey @kunal-parsewar Can you confirm the chat version https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch ? Are you using the helm chart from above provided links? Also just curious any reason to use 1.2.3? Please try with the latest chart There are multiple options available https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch to leverage the sysctl containers. Thank you

[kunal-parsewar]

@prudhvigodithi i have tried with the new helm chart as per your suggestion but still facing the same issue of "Only ReadOnly file system is allowed ". May i know the reasons behind the same.

[prudhvigodithi]

@kunal-parsewar can you share the full error log? which component is causing this ? (pod, statefulset etc)? Can you also share the k8s version? When you added gatekeeper policies, can you also share what are these policies? Can you also check the statefulset if readOnlyRootFilesystem: true is set to true? https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch/values.yaml#L292 Thank you

[kunal-parsewar]

Hi @prudhvigodithi , sorry for the late reply . Please find below error that we were receiving.

[4:14 PM] Bhanu M ERROR: [allow-privilege-escalation] Privilege escalation container is not allowed: fsgroup-volume29s Warning FailedCreate statefulset/opensearch-cluster-master create Pod opensearch-cluster-master-0 in StatefulSet opensearch-cluster-master failed error: admission webhook "validation.gatekeeper.sh" denied the request: [allow-privilege-escalation] Privilege escalation container is not allowed: fsgroup-volume[allow-read-only-root-filesystem] only read-only root filesystem container is allowed: fsgroup-volume

Sorry , I have not set the gatekeeper policies and we do not have access to that.

And we have also tried with this parameter readOnlyRootFilesystem: true but no luck , even we set it to false and tried , still same error.

Kubernetes version on Azure : 1.23.8

Thanks.

[peterzhuamazon]

Hi @kunal-parsewar @prudhvigodithi is it possible for you to try on any other cluster for the deployment? Such as using kind/minikube on your local machine, or other similar online offerings to ensure it is actually an issue with the chart not the kube cluster.

Thanks.

[kunal-parsewar]

@peterzhuamazon i have tried installing on guest OS k8s cluster ( Virtual Box ) , it installed successfully without any issue. Our company environment contains gatekeeper policies that is restricting the installation with the mentioned errors.

Even I have tried the latest version of OpenSearch 2.3

Thanks , Kunal

[kunal-parsewar]

HI Team,

Can anyone help on this?

Regards, Kunal

[smlx]

[allow-privilege-escalation] Privilege escalation container is not allowed: fsgroup-volume

This is the init container which may not be necessary for your environment. --set=persistence.enableInitChown=false should get you past this error since it disables the init container.

[kunal-parsewar]

Thanks @smlx , let me try with that.

Regards, Kunal