OpenSearch cluster is failing with custom admin user/password

rameshar16 commented 5 months ago

[2024-06-10T09:39:10,009][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cr-bootstrap-0] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6d58a6ec] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security) [2024-06-10T09:39:10,009][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cr-bootstrap-0] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6d58a6ec] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security) [2024-06-10T09:39:10,075][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin) [2024-06-10T09:39:10,077][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin) [2024-06-10T09:39:10,078][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin) [2024-06-10T09:39:10,081][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin) [2024-06-10T09:39:12,575][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin) [2024-06-10T09:39:12,577][ERROR][o.o.s.a.BackendRegistry ] [opensearch-cr-bootstrap-0] Not yet initialized (you may need to run securityadmin)

rameshar16 commented 5 months ago

Hi Team,

I am facing the above mentioned issue after I added the custom admin credentials.

=========

opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    version: 2.3.0
    serviceName: "my-first-cluster"
    drainDataNodes: true
    setVMMaxMapCount: true
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    service:
      type: NodePort
    tolerations:
    - effect: NoSchedule
      key: "dedicated"
      operator: "Equal"
      value: "opensearch"
    nodeSelector: {"eks.amazonaws.com/nodegroup": "opensearch-cluster"}
    enable: true
    replicas: 1
    version: 2.3.0
    tls:
      enable: true
      generate: false
      secret:
        name: ssl-secret
      caSecret:
        name: ca-secret
    resources:
      requests:
        memory: "2Gi"
        cpu: "2000m"
      limits:
        memory: "2Gi"
        cpu: "2000m"
  nodePools:
    - component: masters
      nodeSelector: {"eks.amazonaws.com/nodegroup": "opensearch-cluster"}
      tolerations:
      - effect: NoSchedule
        key: "dedicated"
        operator: "Equal"
        value: "opensearch"
      diskSize: "10Gi"
      replicas: 3
      pdb:
        enable: true
        minAvailable: 3
      roles:
        - "cluster_manager"
      resources:
        requests:
          memory: "2Gi"
          cpu: "2000m"
        limits:
          memory: "2Gi"
          cpu: "2000m"
      persistence:
        pvc:
          storageClass: opensearch  # Set the name of the storage class to be used
          accessModes: # You can change the accessMode
          - ReadWriteOnce
    - component: nodes
      nodeSelector: {"eks.amazonaws.com/nodegroup": "opensearch-cluster"}
      tolerations:
      - effect: NoSchedule
        key: "dedicated"
        operator: "Equal"
        value: "opensearch"
      replicas: 3
      pdb:
        enable: true
        maxUnavailable: 2
      diskSize: "10Gi"
      jvm: -Xmx1024M -Xms1024M
      resources:
        requests:
          memory: "2Gi"
          cpu: "2000m"
        limits:
          memory: "2Gi"
          cpu: "2000m"
      roles:
        - "data"
      persistence:
        pvc:
          storageClass: opensearch  # Set the name of the storage class to be used
          accessModes: # You can change the accessMode
          - ReadWriteOnce
    - component: coordinators
      nodeSelector: {"eks.amazonaws.com/nodegroup": "opensearch-cluster"}
      tolerations:
      - effect: NoSchedule
        key: "dedicated"
        operator: "Equal"
        value: "opensearch"
      replicas: 3
      diskSize: "10Gi"
      pdb:
        enable: true
        maxUnavailable: 2
      resources:
         requests:
            memory: "1Gi"
            cpu: "500m"
         limits:
            memory: "1Gi"
            cpu: "500m"
      roles:
        - "ingest"
      persistence:
        pvc:
          storageClass: opensearch  # Set the name of the storage class to be used
          accessModes: # You can change the accessMode
          - ReadWriteOnce
  security:
    config:
      securityConfigSecret:
        name: securityconfig-secret
      adminCredentialsSecret:
        name: admin-credentials-secret
    tls:
      transport:
        generate: true
      http:
        generate: true

======

apiVersion: v1
kind: Secret
metadata:
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: YWRtaW4=
  # admin123
  password: YWRtaW4=

=====

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2a$12$aPsDxUBoupiBjwZRWPvOWu17mc4XVeHKxHiAv3IkvlNbz7n38ItqG"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$aPsDxUBoupiBjwZRWPvOWu17mc4XVeHKxHiAv3IkvlNbz7n38ItqG"
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

rameshar16 commented 5 months ago

Hi All, Please help me on the above issue.

Thank you,

stephen-crawford commented 5 months ago

[Triage] Hi @rameshar16, thank you for filing this issue. Since this looks like a configuration question, it is recommended you ask over on the OpenSearch forum: https://forum.opensearch.org/. You will be able to get better advice on your configuration there.

cwperks commented 5 months ago

@rameshar16 Can you also share opensearch.yml file?

plugins.security.allow_default_init_securityindex must be set to true to source the security index from yaml files on cluster creation.

opensearch-project / security

OpenSearch cluster is failing with custom admin user/password #4422