[BUG] Opensearch dashboards fails to show trace anlytics dashboard for user with limited permissions

[JustinasKO]

Describe the bug user with limited permissions gets 403 when entering app/observability-dashboards#/trace_analytics/home Error on server side:

{
    "type": "response",
    "@timestamp": "2022-06-21T12:23:32Z",
    "tags": [],
    "pid": 1,
    "method": "post",
    "statusCode": 403,
    "req": {
        "url": "/api/observability/trace_analytics/indices",
        "method": "post",
        "headers": {
            "x-forwarded-for": "MASKED",
            "x-forwarded-proto": "https",
            "x-forwarded-port": "443",
            "host": "masked.com",
            "x-amzn-trace-id": "Root=1-62b1b844-41ace150065b5dad2cb24be9",
            "content-length": "0",
            "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
            "content-type": "application/json",
            "sec-ch-ua-mobile": "?0",
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
            "osd-version": "2.0.1",
            "sec-ch-ua-platform": "\"macOS\"",
            "accept": "*/*",
            "origin": "https://masked.com",
            "sec-fetch-site": "same-origin",
            "sec-fetch-mode": "cors",
            "sec-fetch-dest": "empty",
            "referer": "https://masked.com/app/observability-dashboards",
            "accept-encoding": "gzip, deflate, br",
            "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
            "securitytenant": "PROD"
        },
        "remoteAddress": "10.135.166.37",
        "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
        "referer": "https://masked.com/app/observability-dashboards"
    },
    "res": {
        "statusCode": 403,
        "responseTime": 18,
        "contentLength": 9
    },
    "message": "POST /api/observability/trace_analytics/indices 403 18ms - 9.0B"
}

Is it a bug or am I missing some permissions? Used permissions are posted in reproducing steps.

To Reproduce Steps to reproduce the behavior:

1. Create role:

   PUT _plugins/_security/api/roles/trace_analytics
   {
   "cluster_permissions" : [
   "cluster:admin/opensearch/observability/create",
   "cluster:admin/opensearch/observability/delete",
   "cluster:admin/opensearch/observability/get",
   "cluster:admin/opensearch/observability/update"
   ],
   "index_permissions" : [
   {
     "index_patterns" : [
       "otel-v1-*",
       ".opensearch-observability"
     ],
     "allowed_actions" : [
       "read",
       "write",
       "search"
     ]
   }
   ],
   "tenant_permissions" : [
   {
     "tenant_patterns" : [
       "PROD",
       "NON-PROD"
     ],
     "allowed_actions" : [
       "opensearch_dashboards_all_write"
     ]
   }
   ]
   }

2. Create user and assign role: trace_analytics
3. Publish some trace analytics data to cluster
4. Try to access app/observability-dashboards#/trace_analytics/home endpoint with new user.

Expected behavior Dashboard is visible nicely with no error

Plugins Standart plugins

Screenshots

Host/Environment (please complete the following information):

• Opensearch 2.0.1 Docker image
• Opensearch Dashboards 2.0.1 Docker image

Additional context If same is done with all_access user trace analytics data is nicely visible.

[JustinasKO]

UPDATE: changing index permissions from: ["read", "write", "search"] to: ["unlimited"] solves the problem.

    "index_permissions" : [
      {
        "index_patterns" : [
          "otel-v1-*",
          ".opensearch-observability"
        ],
        "allowed_actions" : [
          "unlimited"
        ]
      }
    ],

Documentation probably needs to be updated not be be confusing. https://opensearch.org/docs/latest/observability-plugin/observability-security/