Kibana should avoid using .watches index directly

[masseyke]

In Elasticsearch 8.0, direct access to system indices was intentionally made much more difficult by Elasticsearch. Kibana uses system indices (.watches and .watcher_history) for the "Watcher" part of Stack Management. It looks like it makes a request to _security/user/has_privileges for the .watches and .watcher_history system indices, and only shows the Watcher link if the user has read access to them. And then it does a search request directly on the .watches index. The problem is that, due to the changes in Elasticsearch in 8.0, users have to add that read privilege to their role using the "allow_restricted_indices" flag like this:

curl -k -u elastic:password -X POST "localhost:9200/_security/role/my_watcher_admin_role?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": [
    {
      "names": [ ".watches", ".watcher_history" ],
      "privileges": ["read"],
      "allow_restricted_indices": true
    }
  ]
}
'

But I don't think that Kibana will let you use that flag (I could be wrong). I think this can now be done without accessing system indices at all. There is now an API for fetching all watches (https://github.com/elastic/elasticsearch/pull/64582) that does not require permission to read any system indices directly. If it can't be done without direct access to those indices then the @elastic/es-data-management team probably needs to add whatever is missing.

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[alisonelizabeth]

@ElenaStoeva can you look into the level of effort to address this one?

[ElenaStoeva]

@ElenaStoeva can you look into the level of effort to address this one?

It seems that the Watcher fetches watches by sending a search request on the .watches index in this function, which I think can easily be modified to use the Query Watch API instead as follows:

function fetchWatches(dataClient: IScopedClusterClient) {
  return dataClient.asCurrentUser.watcher.queryWatches();
}

After this, some additional changes are needed to process the response as the Json fields in this response are slightly different from what is returned from the search request to the .watches index.

What concerns me is the part where Watcher fetches a history item by its id. Here, it sends a search request to the .watcher_history index and, since we don't want this, I was thinking what Watcher API we can use instead (@masseyke do you have any suggestion?). I considered using the Get watch API which we can use to retrieve a watch by its id, but I'm not sure if this is appropriate as the fetch function is supposed to return a response from which we can extract a WatchHistoryItem object. @alisonelizabeth, since I am not really familiar with watch history items and what we use them for, do you know how they are related to watches and whether we can convert information from a watch into a watch history item?

[alisonelizabeth]

Thanks @ElenaStoeva for looking into this.

It seems that the Watcher fetches watches by sending a search request on the .watches index in this function, which I think can easily be modified to use the Query Watch API instead as follows:

👍 let's move forward with this change.

What concerns me is the part where Watcher fetches a history item by its id. Here, it sends a search request to the .watcher_history index and, since we don't want this, I was thinking what Watcher API we can use instead (@masseyke do you have any suggestion?).

Yes, good point. I don't think we'll be able to replace this with the current set of ES APIs. Can you open up a separate issue to track this? @masseyke any guidance on this would be appreciated.

[masseyke]

I'm sorry I completely missed the questions on this. We don't currently have an API that reads from the .watcher_history index. It had originally been thought of as something for expert users to go to for troubleshooting. We would need to add that if it's something you need. Watcher is in a semi-deprecated state though, so probably worth a discussion @tylerperk @dakrone.

[dakrone]

The .watcher_history index is a regular (albeit hidden) index, as far as I can tell. Only the .watches index is a system index which requires the special APIs to utilize directly.

[yuliacech]

After a discussion with @ElenaStoeva, we thought that using the Query Watch API for pagination would be a performance improvement, instead of loading all existing watches into the browser. That is currently not possible because the Query Watch API filters and sorts only by the watch id. @masseyke Do you think it would be possible to add more watch properties like state, when it last met condition etc to filtering and sorting?

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)