search

oracle / oracle-database-operator

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.
Universal Permissive License v1.0
141 stars 45 forks source link

DB can not be created on a normal namespace in OpenShift (security!) #100

Closed rbaumgar closed 4 months ago

rbaumgar commented 6 months ago

When I try to deploy the xe-sample to a namespace like "oracle". Operator is not able to create deployment/pod.

runAsUser=54321 is not allowed by default.

2024-05-17T13:12:15Z    INFO    singleinstancedatabase-resource default {"name": "xedb"}
2024-05-17T13:12:15Z    INFO    singleinstancedatabase-resource validate create {"name": "xedb"}
2024-05-17T13:12:15Z    INFO    controllers.database.SingleInstanceDatabase Reconcile requested
2024-05-17T13:12:15Z    INFO    singleinstancedatabase-resource default {"name": "xedb"}
2024-05-17T13:12:16Z    INFO    singleinstancedatabase-resource validate update {"name": "xedb"}
2024-05-17T13:12:16Z    INFO    singleinstancedatabase-resource validate create {"name": "xedb"}
2024-05-17T13:12:16Z    INFO    controllers.database.SingleInstanceDatabase Entering reconcile validation
2024-05-17T13:12:16Z    INFO    controllers.database.SingleInstanceDatabase Completed reconcile validation
2024-05-17T13:12:16Z    INFO    controllers.database.SingleInstanceDatabase Creating a new PVC  {"createPVC Datafiles-Vol": {"name":"xedb","namespace":"oracle"}, "PVC.Namespace": "oracle", "PVC.Name": "xedb"}
2024-05-17T13:12:16Z    INFO    No xedb Pod is Ready    {"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}}
2024-05-17T13:12:16Z    INFO    xedb Pods Available ( Other Than Ready Pod )    {"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}, " Names :": []}
2024-05-17T13:12:16Z    INFO    Total No Of xedb PODS   {"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}, "Count": 0}
2024-05-17T13:12:16Z    INFO    controllers.database.SingleInstanceDatabase Replica Info    {"createPods": {"name":"xedb","namespace":"oracle"}, "Found": 0, "Required": 1}
2024-05-17T13:12:16Z    INFO    controllers.database.SingleInstanceDatabase Creating a new xedb POD {"createPods": {"name":"xedb","namespace":"oracle"}, "POD.Namespace": "oracle", "POD.Name": "xedb-5qt1e"}
2024-05-17T13:12:16Z    ERROR   controllers.database.SingleInstanceDatabase Failed to create new xedb POD   {"createPods": {"name":"xedb","namespace":"oracle"}, "pod.Namespace": "oracle", "POD.Name": "xedb-5qt1e", "error": "pods \"xedb-5qt1e\" is forbidden: unable to validate against any security context constraint: [provider \"anyuid\": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{54321}: 54321 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 54321: must be in the ranges: [1000700000, 1000709999], provider restricted-v2: .containers[0].runAsUser: Invalid value: 54321: must be in the ranges: [1000700000, 1000709999], provider restricted-v2: .containers[0].capabilities.add: Invalid value: \"SYS_NICE\": capability may not be added, provider \"restricted\": Forbidden: not usable by user or serviceaccount, provider \"nonroot-v2\": Forbidden: not usable by user or serviceaccount, provider \"nonroot\": Forbidden: not usable by user or serviceaccount, provider \"hostmount-anyuid\": Forbidden: not usable by user or serviceaccount, provider \"machine-api-termination-handler\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork-v2\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork\": Forbidden: not usable by user or serviceaccount, provider \"hostaccess\": Forbidden: not usable by user or serviceaccount, provider \"hostpath-provisioner\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]"}
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).createPods
    /workspace/controllers/database/singleinstancedatabase_controller.go:2151
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).createOrReplacePods
    /workspace/controllers/database/singleinstancedatabase_controller.go:1915
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).Reconcile
    /workspace/controllers/database/singleinstancedatabase_controller.go:189
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227
rbaumgar commented 6 months ago

How can I remove runAsUser property?

yunus-qureshi commented 6 months ago

@rbaumgar for openshift envs, you must apply this yaml

https://github.com/oracle/oracle-database-operator/blob/main/config/samples/sidb/openshift_rbac.yaml

and specify the service account name "sidb-sa" in the SIDB yaml

rbaumgar commented 6 months ago

This might be a workaround, but is never a solution. Every normal pod has to run with an arbitrary uid. Sorry, a database is a normal pod and does not require special security requirements. You will find much more information on this and several other links https://developers.redhat.com/articles/2021/11/11/best-practices-building-images-pass-red-hat-container-certification#

yunus-qureshi commented 6 months ago

Agreed. The latest v1.1.0 has an attribute called setWritePermissions. Set it to false

https://github.com/oracle/oracle-database-operator/blob/main/config/samples/sidb/singleinstancedatabase.yaml

rbaumgar commented 6 months ago

does not work on singleinstancedatabase_express.yaml

Invalid value: []int64{54321}: 54321 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 54321: must be in the ranges: ... 

spec:
  adminPassword:
    keepSecret: true
    secretKey: oracle_pwd
    secretName: xedb-admin-secret
  createAs: primary
  edition: express
  image:
    prebuiltDB: true
    pullFrom: 'container-registry.oracle.com/database/express:latest'
  pdbName: XEPDB1
  persistence:
    accessMode: ReadWriteOnce
    setWritePermissions: false
    size: 50Gi
    storageClass: oci-bv
  replicas: 1
  sid: XE
yunus-qureshi commented 6 months ago

@rbaumgar also set the attribute prebuiltDB to false

andbos commented 6 months ago

Hi,

When I try to apply openshift_rbac.yaml I get the following error:

$ oc apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/config/samples/sidb/openshift_rbac.yaml
serviceaccount/sidb-sa created
role.rbac.authorization.k8s.io/use-sidb-scc created
rolebinding.rbac.authorization.k8s.io/use-sidb-scc created
error: resource mapping not found for name: "sidb-scc" namespace: "default" from "https://raw.githubusercontent.com/oracle/oracle-database-operator/main/config/samples/sidb/openshift_rbac.yaml": no matches for kind "SecurityContextConstraints" in version "v1"
ensure CRDs are installed first

Installation of the operator went fine:

$ oc apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml
namespace/oracle-database-operator-system created
customresourcedefinition.apiextensions.k8s.io/autonomouscontainerdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabasebackups.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabaserestores.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/cdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/databaseobservers.observability.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dataguardbrokers.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dbcssystems.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/oraclerestdataservices.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/pdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/shardingdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/singleinstancedatabases.database.oracle.com created
role.rbac.authorization.k8s.io/oracle-database-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-proxy-rolebinding created
service/oracle-database-operator-controller-manager-metrics-service created
service/oracle-database-operator-webhook-service created
certificate.cert-manager.io/oracle-database-operator-serving-cert created
issuer.cert-manager.io/oracle-database-operator-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-validating-webhook-configuration created
deployment.apps/oracle-database-operator-controller-manager created

$ oc -n oracle-database-operator-system get pods
NAME                                                           READY   STATUS    RESTARTS   AGE
oracle-database-operator-controller-manager-7f84b7dc4b-994lm   1/1     Running   0          18s
oracle-database-operator-controller-manager-7f84b7dc4b-t5j7r   1/1     Running   0          18s
oracle-database-operator-controller-manager-7f84b7dc4b-twf7d   1/1     Running   0          18s
rbaumgar commented 6 months ago

@andbos this works only on OpenShift. Openshift has an SCC object:

$ oc get crd securitycontextconstraints.security.openshift.io -o yaml|grep storedVersion -A2
  storedVersions:
  - v1
andbos commented 6 months ago

Yes, started testing in OpenShift.

$ oc version
Client Version: 4.14.11
Kustomize Version: v5.0.1
Server Version: 4.14.12
Kubernetes Version: v1.27.10+28ed2d7

The instance was installed properly anyway...

$ oc -n default get singleinstancedatabase
NAME            EDITION      STATUS    ROLE      VERSION      CONNECT STR                                                                            TCPS CONNECT STR   OEM EXPRESS URL
sinchdb11rhos   Enterprise   Healthy   PRIMARY   21.3.0.0.0   605682735.eu-west-1.elb.amazonaws.com:1521/RHOSDB11   Unavailable        https://605682735.eu-west-1.elb.amazonaws.com:5500/em

No errors in the operator logs.

rbaumgar commented 6 months ago

Oh, I see. The SCC is completely wrong formated and the api version is wrong. :-(

IshaanDesai45 commented 4 months ago

@rbaumgar updated the steps to deploy a sidb in a normal namespace/project in openshift . kindly check the PR above

rbaumgar commented 4 months ago

@IshaanDesai45 I created a new NS oracle. updated the deployment of the operator. created a new sid. nothing happens

operator.log```

2024-07-04T08:40:28Z INFO singleinstancedatabase-resource default {"name": "freedb"} 2024-07-04T08:40:28Z INFO singleinstancedatabase-resource validate create {"name": "freedb"} W0704 08:40:52.733694 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list v1alpha1.PDB: pdbs.database.oracle.com is forbidden: User "system:serviceaccount:oracle-database-operator-system:default" cannot list resource "pdbs" in API group "database.oracle.com" in the namespace "oracle" E0704 08:40:52.733826 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch v1alpha1.PDB: failed to list *v1alpha1.PDB: pdbs.database.oracle.com is forbidden: User "system:serviceaccount:oracle-database-operator-system:default" cannot list resource "pdbs" in API group "database.oracle.com" in the namespace "oracle"

IshaanDesai45 commented 4 months ago

The pdb controller is causing this issue. Can you tell me how you are deploying the operator in the namespaced-scope or the cluster-scope

rbaumgar commented 4 months ago

I am using namespace based installation and added your newly created openshift-rbac.

rbaumgar commented 4 months ago

BTW it is a bad design when the operator runs with SA default and has such a rolebinding. should be a nondefault SA.

IshaanDesai45 commented 4 months ago

I am using namespace based installation and added your newly created openshift-rbac.

for using namespace based installation did you also apply the file /rbac/default-ns-rolebinding.yaml with the corresponding namespace ?

IshaanDesai45 commented 4 months ago

BTW it is a bad design when the operator runs with SA default and has such a rolebinding. should be a nondefault SA.

You mean the operator pods that is currently using serviceaccount:oracle-database-operator-sytem:default should use serviceaccount:oracle-database-operator-system:

rbaumgar commented 4 months ago

yes

rbaumgar commented 4 months ago

the problem is fixed, typo when applying rbac/default-ns-rolebinding.yaml, therefor I recommended a different approach. Having an environment variable for the namespace would allow to apply the same file for multiple namespaces.

IshaanDesai45 commented 4 months ago

@rbaumgar we plan to add support of helm charts for the very purpose that user wouldn't need to go and manually change the config/deployment files. So when that is published this your problem of changing the yaml files would be solved