[BUG] Opensearch does not get deployed

sfisli commented 3 months ago

What is the bug?

the opensearch cluster does not get deployed.

What is the expected behavior?

opensearch cluster up and running ( nodes and dashbaord)

What is your host/environment?

Bare Metal Kubernetes v1.26.7

Do you have any additional context?

Operator-values.yaml :

nameOverride: ""
fullnameOverride: ""
domain: monitoring

nodeSelector: {}
tolerations: []
securityContext:
  runAsNonRoot: true
manager:
  securityContext:
    allowPrivilegeEscalation: false
  extraEnv: []
  resources:
    limits:
      cpu: 200m
      memory: 500Mi
    requests:
      cpu: 100m
      memory: 350Mi

  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 8081
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /readyz
      port: 8081
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  # Set this to false to disable the experimental parallel recovery in case you are experiencing problems
  parallelRecoveryEnabled: true

  image:
    repository: opensearchproject/opensearch-operator
    ## tag default uses appVersion from Chart.yaml, to override specify tag tag: "v1.1"
    tag: ""
    pullPolicy: "Always"

  ## Optional array of imagePullSecrets containing private registry credentials
  imagePullSecrets: []
  # - name: secretName

  dnsBase: cluster.local

  # Log level of the operator. Possible values: debug, info, warn, error
  loglevel: info

  # If a watchNamespace is specified, the manager's cache will be restricted to
  # watch objects in the desired namespace. Defaults is to watch all namespaces.
  watchNamespace:

# Install the Custom Resource Definitions with Helm
installCRDs: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Override the service account name. Defaults to opensearch-operator-controller-manager
  name: ""

kubeRbacProxy:
  enable: true
  securityContext:
    # allowPrivilegeEscalation: false
  resources:
    limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 25m
      memory: 25Mi

  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  image:
    repository: "gcr.io/kubebuilder/kube-rbac-proxy"
    tag: "v0.15.0"

opensearch-cluster.yaml:


apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: monitoring
spec:
  security:
     tls:
       transport:
         generate: true
         perNode: true
       http:
          generate: true
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.13.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.13.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 2
      diskSize: "10Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

security-config.yaml :

Waiting to connect to the cluster

Operator logs:

For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187

===============================================

I0614 14:31:32.316748       1 kube-rbac-proxy.go:284] Valid token audiences: 
I0614 14:31:32.316848       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0614 14:31:42.815130       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0614 14:31:42.815173       1 kube-rbac-proxy.go:490] Starting TCP socket on 0.0.0.0:10443
I0614 14:31:42.815651       1 kube-rbac-proxy.go:497] Listening securely on 0.0.0.0:10443 for proxy endpoints
I0614 14:31:42.815727       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443

nilushancosta commented 3 months ago

@sfisli , few questions.

What is the operator and OpenSearch version you are running?
Is the operator in a running and ready state? (Based on kubectl get pods)
Also the logs you added are from a different container in the operator pod. Can you send the logs of the operator-controller-manager container?
And are there any details in the status section when you run kubectl describe opensearchcluster?

sfisli commented 3 months ago

2.6.0

adeiz-opensearch-cluster-securityconfig-update-2sbvg           1/1     Running   0             3m38s
opensearch-operator-controller-manager-596cdf4cfb-r2lzs        2/2     Running   0             5m34s


{"level":"info","ts":"2024-06-18T08:33:23.829Z","msg":"Starting workers","controller":"opensearchrole","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchRole","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.830Z","msg":"Starting workers","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchcomponenttemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchComponentTemplate","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:29.022Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"e74b182a-6ce9-41a1-b8b6-98ce6a9e732d","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.049Z","msg":"Start reconcile - Phase: PENDING","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"e74b182a-6ce9-41a1-b8b6-98ce6a9e732d","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.056Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.081Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","interface":"transport"}
{"level":"info","ts":"2024-06-18T08:33:35.826Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-ca","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:24.975Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-transport-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:24.975Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","interface":"http"}
{"level":"info","ts":"2024-06-18T08:34:28.945Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-http-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:28.946Z","msg":"admin cert does not exist, creating","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660"}
{"level":"info","ts":"2024-06-18T08:34:56.952Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-admin-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:57.056Z","msg":"Starting securityconfig update job","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660"}
{"level":"info","ts":"2024-06-18T08:34:57.068Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-securityconfig-update","namespace":"monitoring","apiVersion":"batch/v1","kind":"Job"}
{"level":"info","ts":"2024-06-18T08:34:57.084Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-config","namespace":"monitoring","apiVersion":"v1","kind":"ConfigMap"}
W0618 08:34:57.096323       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231: failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field RelabelConfig.items.spec.endpoints.relabelings.targetLabel of type string
E0618 08:34:57.096364       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231: Failed to watch *v1.ServiceMonitor: failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field RelabelConfig.items.spec.endpoints.relabelings.targetLabel of type string

4.

Name: adeiz-opensearch-cluster Namespace: monitoring Labels: Annotations: API Version: opensearch.opster.io/v1 Kind: OpenSearchCluster Metadata: Creation Timestamp: 2024-06-18T08:33:29Z Finalizers: Opster Generation: 2 Resource Version: 91823508 UID: 12642e55-ab3d-4784-99e4-1cd1b92f21d3 Spec: Bootstrap: Resources: Conf Mgmt: Dashboards: Enable: true Opensearch Credentials Secret: Name: admin-credentials-secret Replicas: 1 Resources: Limits: Cpu: 200m Memory: 512Mi Requests: Cpu: 200m Memory: 512Mi Service: Type: ClusterIP Tls: Ca Secret: Enable: true Generate: true Secret: Version: 2.13.0 General: Http Port: 9200 Monitoring: Service Name: adeiz-opensearch-cluster Set VM Max Map Count: true Version: 2.13.0 Init Helper: Resources: Node Pools: Component: nodes Disk Size: 10Gi Replicas: 3 Resources: Requests: Cpu: 1 Memory: 2Gi Roles: cluster_manager data Security: Config: Admin Credentials Secret: Name: admin-credentials-secret Admin Secret: Security Config Secret: Name: securityconfig-secret Update Job: Resources: Tls: Http: Ca Secret: Generate: true Secret: Transport: Ca Secret: Generate: true Per Node: true Secret: Status: Components Status: Phase: RUNNING Events: Type Reason Age From Message

Normal Security 2m57s containerset-controller Starting to securityconfig update job

prudhvigodithi commented 3 months ago

[Triage] Hey @sfisli to reach quorum please use atleast 3 replicas, for running cluster as single node here is the workaround from similar issue in past. @nilushancosta @swoehrl-mw @getsaurabh02 @salyh

sfisli commented 3 months ago

@prudhvigodithi same issue, new yaml:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: monitoring
spec:
  security:
     tls:
       transport:
         generate: true
         perNode: true
       http:
          generate: true
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.13.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.13.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 3
      diskSize: "5Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

prudhvigodithi commented 3 months ago

Hey @sfisli can you share your admin-credentials-secret and securityconfig-secret ? I can exactly replicate your config on my local cluster and see where is the error. Thank you

sfisli commented 3 months ago

@prudhvigodithi securityconfig-secret.yaml :


apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$10$wloRN4tgTE0YZbNoklMasOsSy6CMplUEHdKgVV3eN8gw5QTQtKCCO"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2y$10$wloRN4tgTE0YZbNoklMasOsSy6CMplUEHdKgVV3eN8gw5QTQtKCCO"
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

admin-credentials-secret:


apiVersion: v1
kind: Secret
metadata:
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: <base64>
  password: <base64>

prudhvigodithi commented 3 months ago

Thanks @sfisli, works for me, I have matched the config keys used in the provided yaml, was able to access the dashboard and cluster, can you please check the following information?

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: pgcluster
  namespace: default
spec:
  security:
    config: 
     securityConfigSecret:
        name: securityconfig-secret
     adminCredentialsSecret:
        name: admin-credentials-secret
    tls:
       http:
         generate: true 
       transport:
         generate: true
         perNode: true
  general:
    setVMMaxMapCount: true
    serviceName: pgcluster
    version: 2.14.0
    drainDataNodes: true
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    version: 2.14.0
    enable: true
    tls:
      enable: true
      generate: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      resources:
         requests:
            memory: "1Gi"
            cpu: "550m"
         limits:
            memory: "1Gi"
            cpu: "550m"
      roles:
        - "data"
        - "cluster_manager"
      persistence:
         emptyDir: {}

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
## admin opassword hash for test "$2y$12$B6GMBQIwOUEV2qtBQrpJL.37MUMp1XkLxCyWzeTH5Q94QxNjw8ng6"
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$12$tS0wrbNssQpVjOXDPrzqdO5phJC/Fmb9fNKSdJ9P2voGK.LNIqLxG"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

apiVersion: v1
kind: Secret
metadata:  
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: YWRtaW4=
  # test
  password: dGVzdA==

Thank you @getsaurabh02

sfisli commented 3 months ago

@prudhvigodithi the config i provided worked on other clusters v1.27.x , do you think the operator isn't compatible with 1.26.x ?

prudhvigodithi commented 3 months ago

@sfisli when you added v1.27.x and v1.26.x may I know what is this version referred to, is it OpenSearch or operator ?

sfisli commented 3 months ago

@prudhvigodithi i'am talking about the k8s cluster.

prudhvigodithi commented 3 months ago

I dont see any restriction within the operator on 1.26.x version of k8s cluster, adding @swoehrl-mw @salyh.

swoehrl-mw commented 3 months ago

@sfisli

I dont see any restriction within the operator on 1.26.x version of k8s cluster

There are no explicit restrictions to k8s version for the operator. We do not test all versions but I know of nothing that would limit it on k8s 1.26.x, every k8s feature the operator uses should be stable and unchanged between these versions.

As a sidenote: 1.26 is EOL for some months already, so you should not use that version anymore anyway.

mvtab commented 2 months ago

Hi, same problem here with a test cluster:

# opensearch operator values
---
fullnameOverride: opensearch

...

# opensearch cluster
---
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-fluentd
  namespace: logging
spec:
  security:
    tls:
      http:
        generate: False
  general:
    httpPort: 9200
    serviceName: opensearch-fluentd
    version: {{ opensearch_version }}
    pluginsList: []
    vendor: opensearch
  dashboards:
    tls:
      enable: False
    version: {{ opensearch_version }}
    enable: True
    replicas: 1
    resources:
      requests:
        memory: "512Mi"
        cpu: "200m"
      limits:
        memory: "512Mi"
        cpu: "200m"
  nodePools:
  - component: masters
    replicas: 3
    resources:
      requests:
        memory: "4Gi"
        cpu: "1000m"
      limits:
        memory: "4Gi"
        cpu: "1000m"
    roles:
    - "data"
    - "cluster_manager"
    persistence:
      pvc:
        accessModes:
        - ReadWriteOnce
...

# k -n logging get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/opensearch-controller-manager-76d984bff-lfkl2   2/2     Running   0          54m

NAME                                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/opensearch-controller-manager-metrics-service   ClusterIP   10.215.163.197   <none>        8443/TCP   54m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/opensearch-controller-manager   1/1     1            1           54m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/opensearch-controller-manager-76d984bff   1         1         1       54m

# kube-rbac-proxy
# ...
# deprecation warnings
# ...
I0731 14:04:07.988267       1 kube-rbac-proxy.go:284] Valid token audiences: 
I0731 14:04:07.988291       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0731 14:04:17.786451       1 kube-rbac-proxy.go:490] Starting TCP socket on 0.0.0.0:10443
I0731 14:04:17.786459       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0731 14:04:17.786689       1 kube-rbac-proxy.go:497] Listening securely on 0.0.0.0:10443 for proxy endpoints
I0731 14:04:17.786718       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443

# operator-controller-manager
# ...
# the following 3 lines repeating
# ...
{"level":"info","ts":"2024-07-31T15:01:23.549Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","cluster":{"name":"opensearch-fluentd","namespace":"logging"}}
{"level":"error","ts":"2024-07-31T15:01:23.562Z","msg":"Not all secrets for http provided","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","error":"missing secret in spec","stacktrace":"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*TLSReconciler).handleHttp\n\t/workspace/pkg/reconcilers/tls.go:527\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*TLSReconciler).Reconcile\n\t/workspace/pkg/reconcilers/tls.go:72\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2024-07-31T15:01:23.562Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","error":"missing secret in spec","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

Apparently some config secret is missing, but as this was a test cluster I was expecting to get something autogenerated, or actually ignored, since I disabled all tls options.

EDIT: Added these lines to the cluster yaml and it's working.

spec:
  security:
    tls:
       http:
         generate: true 
       transport:
         generate: true
         perNode: true

opensearch-project / opensearch-k8s-operator