[BUG] Error - Invalid username/password for Opensearch dashboard for a valid user when clientauth_mode is set to "REQUIRE"

[nitinjagjivan]

Describe the bug

For security, I want to make enable client certificate authentication mandatory on the OpenSearch nodes. Following Client certificate authentication - OpenSearch documentation 3 I set:

plugins.security.ssl.http.clientauth_mode: "REQUIRE"

The OpenSearch nodes are able to communicate, and I can make API calls using client certs and the admin username/password. The Dashboard can’t connect. Has anyone got this working?

To Reproduce Deploy opensearch and opensearch-dashboards with below settings:

opensearch.yml

cluster.name: my-cluster
network.host: 0.0.0.0

plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: certs/opensearch-transport.crt.pem
        pemkey_filepath: certs/opensearch-transport.key
        pemtrustedcas_filepath: certs/chain-ca.pem
        enforce_hostname_verification: false
        pemkey_password: mypassword
      http:
        enabled: true
        enable_openssl_if_available: true
        pemcert_filepath: certs/opensearch-http.crt.pem
        pemkey_filepath: certs/opensearch-http.key
        pemtrustedcas_filepath: certs/chain-ca.pem
        pemkey_password: mypassword
        clientauth_mode: REQUIRE
    allow_unsafe_democertificates: false
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - "CN=opensearch-admin, OU=CES, O=HCL, C=US"
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]

opensearch-dashboard.yml

logging:
    verbose: false
server:
    name: dashboards
    host: 0.0.0.0
    basePath: "/dashboards"
    ssl:
        enabled: true
        key: certs/elasticstack.key.pem
        certificate: certs/elasticstack.cert.pem
opensearch:
    hosts: https://opensearch-master-node:9200
    requestTimeout: 300000
    username: adminq
    password: mypassword
    ssl:
        certificateAuthorities: certs/chain-ca.pem
        certificate: certs/opensearch-admin.crt.pem
        key: certs/opensearch-admin.key
        verificationMode: none
        alwaysPresentCertificate: true
        keyPassphrase: mypassword

Expected behavior opnesearch-dashboard login page should accept existing username password.

OpenSearch Version v2.9.0

Dashboards Version v2.9.0

Environment deployed using helm

env:
        - name: OPENSEARCH_HOSTS
          value: https://opensearch-master-node:9200
        - name: SERVER_HOST
          value: 0.0.0.0
        - name: OPENSEARCH_USERNAME
          value: adminq
        - name: OPENSEARCH_PASSWORD
          value: mypassword
        - name: KEY_PASSPHRASE
          value: mypassword

[derek-ho]

[Triage] @cwperks will take a look and follow up here if there is an actual bug or if there is a misconfiguration.

[cwperks]

I think I see what the problem is. I just tried to configure this using certs that were not part of the admin dn list (i.e. plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']) and received this error:

server    log   [20:48:17.181] [error][plugins][securityDashboards] StatusCodeError: Authorization Exception
    at respond (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 403,
  displayName: 'AuthorizationException',
  path: '/_plugins/_security/tenantinfo',
  query: {},
  body: undefined,
  statusCode: 403,
  response: '',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
Unhandled Promise rejection detected:

StatusCodeError: Authorization Exception
    at respond (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {

The problem is how its authorized here, though I am not sure yet why its trying to authorize as the cert user rather than the internal user which is the one making the call.

[cwperks]

I was able to get around the issue by change the order of the authenticators so that the clientcert authenticator came after basic authenticator.

@nitinjagjivan Does switching the order of the authenticators help?

[nitinjagjivan]

@cwperks, I tried changing the order but unfortunetly it didn't work.