JordanBarnartt commented 2 weeks ago

What is the bug? We have configured Opensearch and Opensearch Dashboards to use OpenID authentication through Duo. However, every 5 minutes (so it seems, right on the dot), the application "refreshes" itself, briefly redirects through OpenID authentication URLs, then returns back to the webpage. This refresh does not occur automatically, but rather when something in the UI is interacted with after the 5 minute timer has passed. This reload can result in lost work as the page loads to whatever it's default state is (ex. if I am creating an index template, the site will reload back into /app/opensearch_index_management_dashboards, losing any unsaved work). This occurs regardless of whether there was user activity in that 5 minute period.

How can one reproduce the bug? Steps to reproduce the behavior:

Configure opensearch-security/config.yml as follows:

authc:
  openid_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: openid
      challenge: false
      config:
        subject_key: username
        roles_key: roles
        openid_connect_url: <our url>
    authentication_backend:
      type: noop
  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users database"
    http_enabled: true
    transport_enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal

Configure opensearch_dashboards.yml as follows:
```
server.host: "0.0.0.0"
```

opensearch_security.auth.type: "openid" opensearch_security.openid.connect_url: opensearch_security.openid.client_id: opensearch_security.openid.client_secret: opensearch_security.openid.base_redirect_url:

I added these four settings as part of troubleshooting the issue. They did not have any effect.

opensearch_security.cookie.ttl: 86400000 opensearch_security.session.ttl: 86400000 opensearch_security.session.keepalive: true opensearch_security.openid.refresh_tokens: true

opensearch.ssl.verificationMode: none



**What is the expected behavior?**
Activity should "reset" the reauthentication timer.  The amount of time before re-authentication should either be configurable or take from a setting provided by the IdP.

**What is your host/environment?**
 - OS: Opensearch running on Ubuntu 20.04, Opensearch Dashboards run via Docker using opensearchproject/opensearch-dashboards image.
 - Version 2.16.0
 - Plugins: Only those included as part of the default configuration.

**Do you have any additional context?**
This problems seems similar to those described in https://forum.opensearch.org/t/opensearch-dashboards-with-openid-auth-frequent-session-timeouts/15893, https://github.com/opensearch-project/security-dashboards-plugin/issues/71, and https://github.com/opensearch-project/security-dashboards-plugin/issues/1522.  However, the various proposed solutions do not work for us.

My plan is to temporarily disable OIDC auth for Dashboards, but I am willing to perform any troubleshooting that may help solve the issue.

derek-ho commented 1 week ago

Hello @JordanBarnartt, can you check this related issue: https://github.com/opensearch-project/security-dashboards-plugin/issues/1966, we fixed a bug around refresh tokens in 2.15. Also, could you provide your IDP settings to see if this could possibly be a mis-configuration on the IDP side of things?

JordanBarnartt commented 1 week ago

Thanks for your attention, @derek-ho. I double-checked that this was still occurring, and it is. I had since enabled dual authentication using basic internal auth and OIDC. I note that after the ~5 minutes is up, an OIDC user is kicked back to the log in screen. This is different from the behaviour when only OIDC auth is enabled, which would cause a reload but would leave the user signed in.

We're using Duo as an IdP. Here are screenshots of the relevant settings:

derek-ho commented 1 week ago

@JordanBarnartt Can you take a look here: https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/#opensearch-security-configuration? I think the issue might be that you have to swap the order of basic auth and openid authC domains in config.yml.

JordanBarnartt commented 1 week ago

Thanks for pointing that out, @derek-ho, I missed the part about the importance of the ordering.

I switched things around, so now my config.yml looks likes:

  authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: username
            roles_key: roles
            openid_connect_url: https://example.com/oidc/<client_id>/.well-known/openid-configuration
        authentication_backend:
          type: noop

However, whenever I attempt to sign in using SSO, I get the response {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}. There's no logs in Opensearch itself, and Dashboards logs just repeat the 401 message.

If, in my config.yml file, I switch the value of the order keys (so basic is 1 and openid is 0), I am able to log in fine. Given this, I don't think it's an issue with our role mappings.

I tried disabling multiple auth in our opensearch_dashboards.yml file to have only openid auth, but that didn't change anything.

Do you have any idea what the issue might be?

derek-ho commented 1 week ago

@JordanBarnartt is your full opensearch_dashboards.yml shared here: https://github.com/opensearch-project/security-dashboards-plugin/issues/2114#issue-2506545856? If so I think you might be missing opensearch.username and opensearch.password, which is what OSD would use for that internal OpenSearch Dashboards server user auth.

JordanBarnartt commented 1 week ago

That the entire YML file, but we're passing the rest of the arguments as environment variables. Here's what the Dashboards process looks like:

/usr/share/opensearch-dashboards/node/bin/node /usr/share/opensearch-dashboards/src/cli/dist --cpu.cgroup.path.override=/ --cpuacct.cgroup.path.override=/ --opensearch.hosts=["https://host1.example.com:9200", "https://host2.example.com:9200", "https://host3.example.com:9200"] --opensearch.password=<our password> --opensearch.username=kibanaserver --server.name=dev-cluster

derek-ho commented 1 week ago

Can you try setting challenge to be false for the basic authc domain?

JordanBarnartt commented 1 week ago

Okay, that solved the "unauthorized" issue. My config.yml file is now:

authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: username
            roles_key: roles
            openid_connect_url: https://example.com/oidc/<client_id>/.well-known/openid-configuration
        authentication_backend:
          type: noop

However, we're back to the original issue, unfortunately. Every 5 minutes, regardless of activity, an OpenID user gets pushed back to the login screen.

demian711 commented 1 week ago

Hello @derek-ho .

Same issue is happening to us using security-dashboard-plugin and opensearch-dashboards version 2.16.0

In our case, we are using Gitlab.

We are getting disconnected every 2 minutes but only being in the "Dev Tools" screen, it is not happening for example in Discover page. It happens sometimes that page is refreshed but not disconnected. But 9/10 times we are getting disconnected.

What is your host/environment?

OS: Kubernetes / Gitlab Version 2.16.0 Plugins: Defaults and repository-s3

Logs:

{"type":"log","@timestamp":"2024-09-11T07:33:35Z","tags":["error","plugins","securityDashboards"],"pid":1,"message":"Error: Response Error: 400 Bad Request
      at internals.Client._shortcut (/usr/share/opensearch-dashboards/plugins/securityDashboards/node_modules/@hapi/wreck/lib/index.js:569:15)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at callTokenEndpoint (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/helper.ts:88:25)
      at OpenIdAuthentication.isValidCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:290:38)
      at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:143:24
      at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:116:22)
      at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
      at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
      at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
      at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) {
    data: {
      isResponseError: true,
      headers: {
        date: 'Wed, 11 Sep 2024 07:33:35 GMT',
        'content-type': 'application/json; charset=utf-8',
        'transfer-encoding': 'chunked',
        connection: 'close',
        'cache-control': 'no-store',
        'content-security-policy': \"base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/ 'self' https://gitlab.com/assets/ blob: data:; connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com https://collector.prd-278964.gl-product-analytics.com; default-src 'self'; font-src 'self'; form-action 'self' https: http:; frame-ancestors 'self'; frame-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/; img-src 'self' data: blob: http: https:; manifest-src 'self'; media-src 'self' data: blob: http: https:; object-src 'none'; report-uri https://new-sentry.gitlab.net/api/4/security/?sentry_key=f5573e26de8f4293b285e556c35dfd6e&sentry_environment=gprd; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.recaptcha.net/ https://apis.google.com https://*.zuora.com/apps/PublicHostedPageLite.do 'nonce-/otKzBietDOjkd8g4PtADw=='; style-src 'self' 'unsafe-inline'; worker-src 'self' https://gitlab.com/assets/ blob: data:\",
        'referrer-policy': 'strict-origin-when-cross-origin',
        vary: 'Origin',
        'www-authenticate': 'Bearer realm=\"Doorkeeper\", error=\"invalid_grant\", error_description=\"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.\"',
        'x-content-type-options': 'nosniff',
        'x-download-options': 'noopen',
        'x-frame-options': 'SAMEORIGIN',
        'x-gitlab-meta': '{\"correlation_id\":\"01J7G0108YQ2277HQ9C88NRPHT\",\"version\":\"1\"}',
      'x-permitted-cross-domain-policies': 'none',
      'x-request-id': '01J7G0108YQ2277HQ9C88NRPHT',
      'x-runtime': '0.093621',
      'x-xss-protection': '0',
      'gitlab-lb': 'haproxy-main-23-lb-gprd',
      'gitlab-sv': 'web-gke-us-east1-d',
      'cf-cache-status': 'DYNAMIC',
      'report-to': '{\"endpoints\":[{\"url\":\"https:\\\\/\\\\/a.nel.cloudflare.com\\\\/report\\\\/v4?s=kmxIHCtraUuUO1WOkylN8gY4OcRI%2FZC2E0sKRM43uMjCbVwrN2D5hHpXbSGLItfPcgZodJ07Pwegqmlix89nXyVxSPjt6caX%2BRTKuEgOcbwWMIyP5FyaifaJofY%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}',
      nel: '{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}',
      'strict-transport-security': 'max-age=31536000',
      'set-cookie': [Array],
      server: 'cloudflare',
      'cf-ray': '8c15f86e3b262c7a-FRA'
    },
    res: IncomingMessage {
        _readableState: [ReadableState],
        _events: [Object: null prototype],
        _eventsCount: 2,
        _maxListeners: undefined,
        socket: [TLSSocket],
        httpVersionMajor: 1,
        httpVersionMinor: 1,
        httpVersion: '1.1',
        complete: true,
        rawHeaders: [Array],
        rawTrailers: [],
        joinDuplicateHeaders: undefined,
        aborted: false,
        upgrade: false,
        url: '',
        method: null,
        statusCode: 400,
        statusMessage: 'Bad Request',
        client: [TLSSocket],
        _consuming: true,
        _dumped: false,
        req: [ClientRequest],
        [Symbol(kCapture)]: false,
        [Symbol(kHeaders)]: [Object],
        [Symbol(kHeadersCount)]: 52,
        [Symbol(kTrailers)]: null,
        [Symbol(kTrailersCount)]: 0

    },
    payload: <Buffer 7b 22 65 72 72 6f 72 22 3a 22 69 6e 76 61 6c 69 64 5f 67 72 61 6e 74 22 2c 22 65 72 72 6f 72 5f 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 68 65 20 ... 163 more bytes>
  },
  isBoom: true,
  isServer: false,
  output: {
      statusCode: 400,
      payload: {
        statusCode: 400,
        error: 'Bad Request',
        message: 'Response Error: 400 Bad Request'

    },
    headers: {}
  }
}"}
{"type":"response","@timestamp":"2024-09-11T07:33:35Z","tags":["access:console"],"pid":1,"method":"post","statusCode":401,"req":{"url":"/api/console/proxy?path=_template&method=GET&dataSourceId=","method":"post","headers":{"host":"log-svc-tst-mv.demo.dev.io","x-request-id":"f028ab2a133a43869fdfc74dc4c1e17a","x-real-ip":"193.159.177.6","x-forwarded-for":"193.159.177.6","x-forwarded-host":"log-svc-tst-mv.demo.dev.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","geo-country-code":"DE","x-forgerock-transactionid":"f028ab2a133a43869fdfc-DB182A0A76E8B5661501D1810203","x-uule-get":"uule=DB182A0A76E8B5661501D18102030303","content-length":"0","sec-ch-ua":"\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\"","sec-ch-ua-platform":"\"macOS\"","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","osd-version":"2.16.0","content-type":"application/json","accept":"*/*","sec-gpc":"1","accept-language":"en-GB,en;q=0.9","origin":"https://log-svc-tst-mv.demo.dev.io","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://log-svc-tst-mv.demo.dev.io/app/dev_tools","accept-encoding":"gzip, deflate, br, zstd","priority":"u=1, i","x-sl-req-uule":"f028ab2a133a43869fdfc-DB182A0A76E8B5661501D1810203"},"remoteAddress":"10.42.19.118","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","referer":"https://log-svc-tst-mv.demo.dev.io/app/dev_tools"},"res":{"statusCode":401,"responseTime":234,"contentLength":9},"message":"POST /api/console/proxy?path=_template&method=GET&dataSourceId= 401 234ms - 9.0B"}
{"type":"log","@timestamp":"2024-09-11T07:33:35Z","tags":["error","plugins","securityDashboards"],"pid":1,"message":"Error: Response Error: 400 Bad Request
      at internals.Client._shortcut (/usr/share/opensearch-dashboards/plugins/securityDashboards/node_modules/@hapi/wreck/lib/index.js:569:15)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at callTokenEndpoint (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/helper.ts:88:25)
      at OpenIdAuthentication.isValidCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:290:38)
      at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:143:24
      at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:116:22)
      at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
      at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
      at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
      at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) {
    data: {
      isResponseError: true,
      headers: {
        date: 'Wed, 11 Sep 2024 07:33:35 GMT',
        'content-type': 'application/json; charset=utf-8',
        'transfer-encoding': 'chunked',
        connection: 'close',
        'cache-control': 'no-store',
        'content-security-policy': \"base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/ 'self' https://gitlab.com/assets/ blob: data:; connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com https://collector.prd-278964.gl-product-analytics.com; default-src 'self'; font-src 'self'; form-action 'self' https: http:; frame-ancestors 'self'; frame-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/; img-src 'self' data: blob: http: https:; manifest-src 'self'; media-src 'self' data: blob: http: https:; object-src 'none'; report-uri https://new-sentry.gitlab.net/api/4/security/?sentry_key=f5573e26de8f4293b285e556c35dfd6e&sentry_environment=gprd; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.recaptcha.net/ https://apis.google.com https://*.zuora.com/apps/PublicHostedPageLite.do 'nonce-DoAsjkXsn5mcBQD5WIMvXg=='; style-src 'self' 'unsafe-inline'; worker-src 'self' https://gitlab.com/assets/ blob: data:\",
        'referrer-policy': 'strict-origin-when-cross-origin',
        vary: 'Origin',
        'www-authenticate': 'Bearer realm=\"Doorkeeper\", error=\"invalid_grant\", error_description=\"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.\"',
        'x-content-type-options': 'nosniff',
        'x-download-options': 'noopen',
        'x-frame-options': 'SAMEORIGIN',
        'x-gitlab-meta': '{\"correlation_id\":\"01J7G01097QSEVNG2RGGWF7QBT\",\"version\":\"1\"}',
      'x-permitted-cross-domain-policies': 'none',
      'x-request-id': '01J7G01097QSEVNG2RGGWF7QBT',
      'x-runtime': '0.089958',
      'x-xss-protection': '0',
      'gitlab-lb': 'haproxy-main-24-lb-gprd',
      'gitlab-sv': 'web-gke-us-east1-b',
      'cf-cache-status': 'DYNAMIC',
      'report-to': '{\"endpoints\":[{\"url\":\"https:\\\\/\\\\/a.nel.cloudflare.com\\\\/report\\\\/v4?s=LmCSqZUBEDKLfg3%2Bh813UqYOso77nVUYoAmqq9yJKm17K%2Bo4yack2cTw7Auk77uJu0nQnGNQrOVQEA5w%2Bs3sVwWqHNdo7FjtK8%2F8Fq%2BLsHwqkyPmYYf1lZZVHm4%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}',
      nel: '{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}',
      'strict-transport-security': 'max-age=31536000',
      'set-cookie': [Array],
      server: 'cloudflare',
      'cf-ray': '8c15f86e4ccc2bd7-FRA'
    },
    res: IncomingMessage {
        _readableState: [ReadableState],
        _events: [Object: null prototype],
        _eventsCount: 2,
        _maxListeners: undefined,
        socket: [TLSSocket],
        httpVersionMajor: 1,
        httpVersionMinor: 1,
        httpVersion: '1.1',
        complete: true,
        rawHeaders: [Array],
        rawTrailers: [],
        joinDuplicateHeaders: undefined,
        aborted: false,
        upgrade: false,
        url: '',
        method: null,
        statusCode: 400,
        statusMessage: 'Bad Request',
        client: [TLSSocket],
        _consuming: true,
        _dumped: false,
        req: [ClientRequest],
        [Symbol(kCapture)]: false,
        [Symbol(kHeaders)]: [Object],
        [Symbol(kHeadersCount)]: 52,
        [Symbol(kTrailers)]: null,
        [Symbol(kTrailersCount)]: 0

    },
    payload: <Buffer 7b 22 65 72 72 6f 72 22 3a 22 69 6e 76 61 6c 69 64 5f 67 72 61 6e 74 22 2c 22 65 72 72 6f 72 5f 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 68 65 20 ... 163 more bytes>
  },
  isBoom: true,
  isServer: false,
  output: {
      statusCode: 400,
      payload: {
        statusCode: 400,
        error: 'Bad Request',
        message: 'Response Error: 400 Bad Request'

    },
    headers: {}
  }
}"}

Thank you, let me know if you need a new Bug to be open.

rayangou commented 3 days ago

@JordanBarnartt Have you tried adding the "scope" in opensearch_dashboards.yml

opensearch_security.openid.scope: "openid email offline_access"

offline_access - is needed for refreshing the token

opensearch-project / security-dashboards-plugin

[BUG] OpenID sessions re-authentication every 5 minutes #2114

I added these four settings as part of troubleshooting the issue. They did not have any effect.