search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.73k stars 24.68k forks source link

System Indices #50251

Open jaymode opened 4 years ago

jaymode commented 4 years ago

This issue serves as an overarching issue for what we call System Indices and details our plan to implement this feature.

As Elasticsearch and the rest of the stack has evolved, there is an increasing reliance on indices as an implementation detail of features. Certain features are critical, like Security, and this implementation detail should not be necessary for a user to know or think about. Unfortunately, this implementation detail has been exposed to users in various ways.

In general, most of the indices that would be considered system indices have names that start with a . and are referred to as dot indices. This is modeled after dot files to suggest that the index is an implementation detail that should not be interacted with. Searches over all indices, such as those done by Kibana by default, include dot indices, while to a user all indices means all indices they created. Snapshots include dot indices by default, but restoring dot indices does not work since the intended index already exists and should be wholly replaced. And upgrades of dot indices are managed with complicated aliases and versioned names, where each dot index has its own versioning and upgrade logic that must be run externally.

Dot indices will be superseded by system indices that are registered through a module or plugin. A special plugin type will be added that defines the system indices that the plugin provides along with the mappings and APIs necessary to access these indices. The existing search API (and other data APIs) will not be able to search a system index. APIs used for monitoring and troubleshooting will continue to operate as they do today and include system indices in their output.

There are some dot indices that do not necessarily fit the mold of a normal system index; instead they store data that the system produces with the intent that users can also query against this data. For these indices we will move to adding a hidden index that will not be resolved by default for wildcards. These indices can be specifically requested or an IndicesOption can be specified so that these indices are not ignored during wildcard resolution. An index may only be marked as hidden at creation time and will be done so through the use of an index setting.

Hidden indices will not may have names that are prefixed by a dot. Hidden indices will not inherit from a global index template.

In order to facilitate the migration from accessing dot indices directly the following phased approach will be taken.

The initial work is the introduction of the infrastructure to define hidden indices and system index plugins with dedicated APIs.

Tasks

The above steps will get us to a point where other teams can begin migrating from direct access to their system index and make use of the new APIs. The majority of the internal (to Elasticsearch) work will still need to be implemented.

System Index Patterns

IngestGeoIpPlugin

KibanaPlugin

AsyncResultsIndexPlugin

EnrichPlugin

Fleet

Logstash

MachineLearning

SearchableSnapshots

Security

Transforms

Watcher

APIs allowed to access system indices

The APIs that we currently plan to allow system index access for are:

elasticmachine commented 4 years ago

Pinging @elastic/es-core-infra (:Core/Infra/Core)

hendrikmuhs commented 3 years ago

What about POST _refresh?

This currently returns the deprecation warning added in #60945. It's odd that even in case I do not specify a concrete name I get the warning. It's unclear for a user what was wrong. Looks like the resolver should filter out system indices.

gwbrown commented 3 years ago

It's odd that even in case I do not specify a concrete name I get the warning.

This is one of the less intuitive effects of deprecating access to system indices - using a total wildcard (*/_all, as POST _refresh does under the hood) results in a deprecation warning because system indices are indeed accessed as part of that request. The best way to prevent the deprecation warning is to specify an index pattern that does not include system indices.

We went back and forth on this behavior in https://github.com/elastic/elasticsearch/pull/60945, but ultimately decided to accept some odd deprecation warnings as the alternative is sometimes not warning when a request will change behavior in 8.0.

Looks like the resolver should filter out system indices.

This is indeed what we plan to do for 8.0 - but we can't yet as there are few pieces of infrastructure we're still working on. In the checklist in the OP, the items above Disallow direct access of system indices need to be addressed before we can start filtering out system indices. At that time, we'll also remove the deprecation warnings from master. Not that these changes will only be made in master, as filtering system indices from requests in 7.x would be a breaking change.

It was suggested on Slack that we could adjust the text of the deprecation warnings if a total wildcard is in use, to clarify things for users. This is possible, but is not trivial as Security obscures the original index pattern before we emit the deprecation warning - there's code in the history of https://github.com/elastic/elasticsearch/pull/60945 to allow changing the behavior of the deprecation warning for total wildcards, but it's a bit hacky.

nemhods commented 3 years ago

Hey, i just got to the bottom of a problem that prevented Fleet Server from working, and it comes back to this issue. I had an index template set up like { "index_patterns": [ "*" ], "order": 10, "settings": { "number_of_shards": "2", "number_of_replicas": "1" } } }

... which also affected the system indices .fleet-actions and .fleet-policies. This resulted in the fleet server not working, logging lines like:

{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error.message":"elastic fail 400:status_exception:wait_for_advance only supports indices with one shard. [shard count: 2]","@timestamp":"2021-06-13T16:38:14.423Z","message":"failed on waiting for global checkpoints advance"}

I manually repaired this by shrinking the affected indices down to one shard again, as well as changing my custom template. This wouldn't have happened if my custom wildcard template was prevented from tampering with system indices.

More here: https://discuss.elastic.co/t/osquery-live-queries-dont-go-through/274428/5

williamrandolph commented 3 years ago

@nemhods Thanks for reporting this issue with Fleet Server and system indexes. That was impressive detective work, helpfully explained, and we appreciate it a lot. I've created a new issue for some targeted work around templates and system indices: #74271

hungnguyen-elastic commented 3 years ago

What is the plan for the .ds-* data stream indices? They are not really system indices but when users try to delete the index that is part of a data stream, Kibana gives this error which seems more scary than it really is

image

gwbrown commented 3 years ago

@hungnguyen-elastic That seems like a Kibana bug. Data stream backing indices are hidden indices, and have been since their introduction.

hungnguyen-elastic commented 3 years ago

Thanks @gwbrown - ticket #106040 was created for this

Fyi @nicpenning and @nmprokop

stefnestor commented 2 years ago

Had pasted reformatted table of system indices. It's not "official" in any sense. Removing to sub-drop-down to avoid confusing folks googling.

before 👋🏼 This Github appears to be a popular [system index](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#system-indices) reference point / collection, so expanding known indices/aliases from description list (still hugely WIP): feature | alias | index --|--|-- elastic endgame | `endgame-${version}` | `endgame-${version}-*` elasticsearch enrich geoip | NULL | `.geoip_databases` elasticsearch enrich | UNK | `.enrich-*` elasticsearch index ilm | `ilm-history-${#}` | `ilm-history*` elasticsearch index search async | NULL | `.async-search` elasticsearch index searchable snapshot | NULL | `.snapshot-blob-cache` elasticsearch ingest transform | `.data-frame-internal-${#}` | `.transform-internal-*` elasticsearch ingest transform | `.transform-notifications-read` | `.transform-notifications-*` elasticsearch node (task/threadPool?) | NULL | `.tasks` ([StackOverFlow](https://stackoverflow.com/questions/54882478/elasticsearch-tasks-index-usage)) at least v5.0 through v17.0 elasticsearch security | `.security-tokens` | `.security-tokens-[0-9]+` elasticsearch security | `.slm-history-${#}` | `.security-[0-9]+` elasticsearch snapshot slm | `.slm-history-${#}` | `.slm-history*` elasticsearch watcher | UNK | `.triggered_watches*` elasticsearch watcher | UNK | `.watches*` kibana analytic machine learning | UNK | `.ml-annotations*` kibana analytic machine learning | UNK | `.ml-anomalies-*` kibana analytic machine learning | UNK | `.ml-config*` kibana analytic machine learning | UNK | `.ml-inference-*` kibana analytic machine learning | UNK | `.ml-meta*` kibana analytic machine learning | UNK | `.ml-notifications*` kibana analytic machine learning | UNK | `.ml-state*` kibana analytic machine learning | UNK | `.ml-stats-*` kibana entSearch | UNK | `.ent-search*` kibana fleet | `.fleet-actions` | `.fleet-actions~(-results*)` kibana fleet | `.fleet-agents` | `.fleet-agents*` kibana fleet | `.fleet-artifacts` | `.fleet-artifacts*` kibana fleet | `.fleet-enrollment-api-keys` | `.fleet-enrollment-api-keys*` kibana fleet | `.fleet-policies-leader` | `.fleet-policies-leader*` kibana fleet | `.fleet-policies` | `.fleet-policies-[0-9]+*` kibana fleet | `.fleet-servers` | `.fleet-servers*` kibana fleet | UNK | `.fleet-actions-results` kibana observ apm | UNK | `.apm-agent-configuration` kibana observ apm | UNK | `.apm-custom-link` kibana reporting | UNK | `.reporting-*` kibana security | NULL | `.kibana_security_session*` kibana siem | `.alerts-security.alerts-${space}` | `.alerts-security.alerts-*`, `.siem-signals-${space}-*` kibana siem | `.items-${space}` | `.items-${space}-*` kibana siem | `.lists-${space}` | `.lists-${space}-*` kibana siem | UNK | `.alerts-observability` kibana | `.kibana-event-log-${version}` | `.kibana-event-log-${version}-*` kibana | `.kibana_task_manager` | `.kibana_task_manager_${version}_*` kibana | `.kibana` | `.kibana_${version}_*` logstash | UNK | `.logstash` UNK | UNK | `.metrics-endpoint`
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

krishhteja commented 6 months ago

Once everything is migrated and access system indices is removed, is it possible to access to system indices with some elevated permissions or using rules? or is it not possible at all?

williamrandolph commented 6 months ago

@krishhteja There are still some back doors for superuser access to system indices, but our intention is that any management of or interaction with a system index can be performed via feature-specific REST APIs. See, for example, the PUT /_security/settings API. If there's something you need to be able to do but we don't have an API for it, we can consider a feature request, though you may want to raise your issue in our discuss forums first.