[Bug]: Clicking Save on Discover tab leads to permissions issue

[atalakey4work]

Describe the bug

After upgrading to OpenSearch and OpenSearch Dashboards version 2.16.0, we get error when we try to save a Discover search.

To reproduce

1. Go to OpenSearch Dashboard Discover
2. Click on 'Save'
3. Inout any title
4. Click on 'Save'

Expected behavior

Discover search is saved with no errors

Screenshots

Host / Environment

OpenSearch and OpenSearch Dashboards version 2.16.0

Additional context

No response

Relevant log output

{"type":"log","@timestamp":"2024-09-22T04:16:47Z","tags":["error","opensearch","data"],"pid":1,"message":"[security_exception]: no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}

{"type":"response","@timestamp":"2024-09-22T04:16:47Z","tags":[],"pid":1,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/search?overwrite=true","method":"post","headers":{"host":"[MASKED]","x-request-id":"cd1b40e738290336e5236bd11c7e11f3","x-real-ip":"11.0.0.5","x-forwarded-for":"[MASKED]","x-forwarded-host":"[MASKED]","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"599","sec-ch-ua":"\"Chromium\";v=\"128\", \"Not;A=Brand\";v=\"24\", \"Google Chrome\";v=\"128\"","sec-ch-ua-platform":"\"[MASKED]\"","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"[MASKED]","osd-version":"2.16.0","content-type":"application/json","accept":"*/*","origin":"https://[MASKED]","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://[MASKED]/app/data-explorer/discover","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9,ar;q=0.8","priority":"u=1, i"},"remoteAddress":"10.244.1.199","userAgent":"[MASKED]","referer":"https://[MASKED]/app/data-explorer/discover"},"res":{"statusCode":403,"responseTime":19,"contentLength":9},"message":"POST /api/saved_objects/search?overwrite=true 403 19ms - 9.0B"}

[gaiksaya]

Transferring this issue to core dashboards repo.

[ashwin-pc]

Looks like an authentication issue. Do you have any additinal details on why you werent able to save the search? Also @opensearch-project/admin can you transfer this to the dashboards security repo since this seems to be an auth related issue.

[cwperks]

@atalakey4work What version are you upgrading from? Was this working prior to upgrade? Are you searching on any index pattern in particular?

[atalakey4work]

@cwperks from version 2.16.0 to version 2.17.0.

Note that I’m using the helm chart.

[cwperks]

What is the index pattern that the search is pertaining to and what plugins are installed on the cluster?

[atalakey4work]

There are no plugins, other than the out of the box plugins if any, installed on the cluster.

The issue occurs with any index pattern.

[cwperks]

[Triage] @atalakey4work Can you provide a little more information about the setup? I am not able to reproduce the issue as described.

[atalakey4work]

@cwperks the issue started after upgrading OpenSearch and OpenSearch Dashboards to version 2.17.0. After the upgrade, I am no longer able to save new searches nor open searches previously saved. I had to downgrade OpenSearch Dashboards to version 2.16.0 to get the saved searches to work but I am still unable to save new searches. I was not able to downgrade OpenSearch as it would not allow me. I suspect it had something to do with the security plugin config so I took the latest copy from the below repo:

https://github.com/opensearch-project/security/tree/main/config

[cwperks]

What index patterns are the saved searches on? I have a suspicion that there's a saved search that includes searching on system indices (system indices always start with a dot .). I will try to replicate the 2.16 to 2.17 upgrade and see what steps would be needed for mitigation. In general, if you have a saved search on a broad index pattern (like *), then it should be narrowed to a more specific index pattern.

[atalakey4work]

@cwperks I was able to resolve the error by updating the below in my opensearch_dashboarfs.yml

Before: opensearch.requestHeadersAllowlist: ["Authorization", "securitytenant"]

After: opensearch.requestHeadersAllowlist: ["Authorization"]

Now save works but whenever I open a saved search the saerch filters are not populated:

Here is a copy of the kibanaSavedObjectMeta.searchSourceJSON

{
  "query": {
    "query": "",
    "language": "kuery"
  },
  "highlightAll": true,
  "version": true,
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "",
        "min_doc_count": 1
      }
    }
  },
  "filter": [
    {
      "meta": {
        "alias": null,
        "negate": false,
        "disabled": false,
        "type": "phrase",
        "key": "kubernetes.namespace_name",
        "params": {
          "query": "nginx-ingress"
        },
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
      },
      "query": {
        "match_phrase": {
          "kubernetes.namespace_name": "nginx-ingress"
        }
      },
      "$state": {
        "store": "appState"
      }
    }
  ],
  "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
}