elastic / curator

Curator: Tending your Elasticsearch indices
Other
3.04k stars 635 forks source link

Trying to snapshot all indices fails as Curator accesses system indices #1683

Open WilliamDEdwards opened 1 year ago

WilliamDEdwards commented 1 year ago

Expected Behavior

When trying to create a snapshot of all indices with Curator, such as with the following config, no error should occur.

---
actions:
  1:
    action: snapshot
    description: >-
      Snapshot selected indices to 'repository' with the snapshot name or name
      pattern in 'name'.  Use all other options as assigned
    options:
      repository: s3-dadup
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
      name:
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      disable_action: False
    filters:
    - filtertype: none

Actual Behavior

Elasticsearch 8.x deprecated direct access to system indices. This causes the following error:

2023-07-08 10:33:48,810 ERROR     Failed to complete action: snapshot.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs formore information. Exception: BadRequestError(400, 'illegal_argument_exception', 'the [indices] parameter includes system indices [.kibana_7.13.3_001, .kibana_7.12.1_001, .kibana_7.14.0_001, .kibana_task_manager_7.17.7_001, .kibana_task_manager_7.14.0_001, .kibana_task_manager_7.12.1_001, .kibana_7.17.6_001, .kibana_task_manager_7.17.1_001, .kibana_7.12.0_001, .kibana_task_manager_7.13.3_001, .kibana_7.15.1_001, .kibana_7.16.3_001, .kibana_7.17.5_001, .kibana_task_manager_7.12.0_001, .security-6-reindexed-for-8, .kibana_task_manager_7.16.3_001, .kibana_7.17.1_001, .kibana_task_manager_7.15.1_001, .kibana_7.17.9_001, .transform-internal-007, .fleet-enrollment-api-keys-7, .kibana_7.17.11_001, .transform-internal-006, .kibana_task_manager_7.17.9_001, .transform-internal-005, .kibana_task_manager_7.17.5_001, .kibana_1-reindexed-for-8, .kibana_task_manager_7.17.11_001, .kibana_task_manager_7.17.3_001, .kibana_7.16.1_001, .kibana_task_manager_7.17.6_001, .kibana_task_manager_7.16.1_001, .kibana_7.17.7_001, .fleet-policies-7, .kibana_7.17.3_001]; to include or exclude system indices from a snapshot, use the [include_global_state] or [feature_states] parameters')

Steps to Reproduce the Problem

  1. Install Curator 8.x
  2. Install Elasticsearch 8.x
  3. Run the snapshot config above

Specifications

Context (Environment)

N/A

Detailed Description

I believe Curator should skip system indices, or make that configurable in case the user created an Elasticsearch user that does have access using the allow_restricted_indices permission. Note that I am using the built-in elastic user.