[DataViews] GetIndices should allow me to get non-hidden indices

[stratoula]

Describe the feature:

Not sure if this is a bug or a feature but is important for ES|QL.

We are suggestiing indices in the from command using this api

const indices = await dataViews.getIndices({
    showAllIndices: false,
    pattern: '*',
    isRollupIndex: () => false,
  });

and then we mark them as hidden if index.name.startsWith('.') but this is wrong. Not all the indices starting with . are hidden so we need a way to receive all the non-hidden (system indices) and remove the wrong check for the dot.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

I created a hidden index

{
  "settings": {
    "index.hidden": true
  },
  "mappings": {
    "properties": {
      "field1": {
        "type": "text"
      },
      "field2": {
        "type": "keyword"
      }
    }
  }
}

POST /my_hidden_index/_doc/1
{
  "field1": "This is some text data",
  "field2": "keyword_value"
}

and removed the index.name.startsWith('.')

when trying out dataViews.getIndices

if was not part of the returned list, so it might be sufficient to remove this check to get all non-hidden indices?

[stratoula]

Hmmm this gives me this list

@ninoslavmiskovic are these indices safe to suggest?

[stratoula]

@kertal these are not being suggested in the Index management page so I am not expecting them to be suggested, right?

[jughosta]

these are not being suggested in the Index management page

They might be on Data Streams tab.

[stratoula]

Not all of them Jul

[kertal]

@kertal these are not being suggested in the Index management page so I am not expecting them to be suggested, right?

I had another look at it, and it seems most of those are tagged alias , e.g. .siem-signals-default

this seems to be an alias for the following index:

.internal.alerts-security.alerts-default-000001

which is appears to be a hidden index?

I'm not sure if this alias should be displayed? In the Index Management, it is not findable?

[stratoula]

It should not be suggested if it is an alias to a hidden index. Can the api work like that? Display only indices / aliases that are not hidden (exactly as the Index management page behaves when the Include hidden indices switch is off)

[kertal]

It should not be suggested if it is an alias to a hidden index. Can the api work like that? Display only indices / aliases that are not hidden (exactly as the Index management page behaves when the Include hidden indices switch is off)

I took a look, but it seems to me, this is currently not supported. If a near term solution is needed a workaround would be for the consumer to filter out those alias to a hidden index. Something like first create a list of not-alias indices, then add alias that point to indices that can be found in the non-alias indices list

[stratoula]

@kertal I would like to avoid this in the client side, the api should handle this for performance reasons. So I will wait for this to be prioritized cc @ninoslavmiskovic

[kertal]

. @kertal I would like to avoid this in the client side, the api should handle this for performance reasons. So I will wait for this to be prioritized cc @ninoslavmiskovic

I do agree that a server side solution is preferable 👍

[ninoslavmiskovic]

++ on not doing it on the client side.

I don't believe it is possible in ES to directly prevent users from creating aliases on hidden indices.

However, as a user, it is possible to be able to control access to hidden indices and alias creation through role-based access control (RBAC). E.g Restrict Index Access:, Aliases Privilege (https://www.elastic.co/guide/en/elasticsearch/reference/current/securing-aliases.html#index-alias-privileges), Custom Roles, Index-Level Security:

Especially the aliases' privileges are interesting

I guess you are a super user @stratoula - perhaps that is why you are able to create aliases with hidden indices.?

[kertal]

@elastic/kibana-management I'm interested in the logic behind "Include hidden indices" on Index Management, so the whole list is returned, and the filter is applied client side, on the "hidden" property, right? how does it work server side. thx!

[sabarasaba]

That is correct, we return all indices or datastreams and the pagination happens client side based on the hidden property as you mentioned. This hidden property is computed based on the settings.index.hidden attribute of that given index, you can find the code that deals with creating that response here.

[kertal]

@sabarasaba thx, this is very helpful!

[kertal]

No 100% sure, but it might be the case that we are already using the same code under the hood, so in theory the hidden property "just" needs to be made available to the consumers in UI https://github.com/elastic/kibana/blob/628994ea3addcca28332ffae81034370a92a6267/src/plugins/data_views/public/services/get_indices.ts#L43-L60

FYI @mattkime (once back from PTO)