search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.43k stars 24.57k forks source link

Get snapshot API with `verbose` = `false` returns an empty `data_streams` object #111145

Open romain-chanu opened 1 month ago

romain-chanu commented 1 month ago

Elasticsearch Version

8.14.3

Installed Plugins

No response

Java Version

bundled

OS Version

N.A

Problem Description

Get snapshot API with verbose = false returns an empty data_streams object.

This is rather confusing and inconsistent with the overall behaviour (e.g feature_states is not reported when verbose = false).

Omitting the data_streams object when verbose = false would be more logical (similar to the feature_states object)

It is also not documented that verbose = true actually returns the data streams and feature states information as well:

(Optional, Boolean) If true, returns additional information about each snapshot such as the version of Elasticsearch which took the snapshot, the start and end times of the snapshot, and the number of shards snapshotted. Defaults to true. If false, omits the additional information.

Steps to Reproduce

{
  "total": 1,
  "remaining": 0,
  "snapshots": [
    {
      "uuid": "PSIEWRmOR1m5EzTiCZJANg",
      "repository": "found-snapshots",
      "state": "SUCCESS",
      "snapshot": "cloud-snapshot-2024.07.22-wpnxv4hmqzqb3vewczcjvq",
      "indices": [
        ".internal.alerts-stack.alerts-default-000001",
        ".kibana_ingest_8.14.3_001",
        ".ds-metrics-fleet_server.agent_status-default-2024.07.21-000001",
        ".slo-observability.sli-v3.2",
        ".internal.alerts-observability.metrics.alerts-default-000001",
        ".ds-metrics-system.process-default-2024.07.21-000001",
        ".fleet-servers-7",
        ".ds-metrics-elastic_agent.filebeat_input-default-2024.07.21-000001",
        ".kibana_task_manager_8.14.3_001",
        ".internal.alerts-observability.apm.alerts-default-000001",
        ".fleet-agents-7",
        ".transform-internal-007",
        ".ds-logs-osquery_manager.result-default-2024.07.21-000001",
        ".ds-metrics-elastic_agent.metricbeat-default-2024.07.21-000001",
        ".apm-source-map",
        ".ds-ilm-history-7-2024.07.21-000001",
        ".kibana_security_solution_8.14.3_001",
        ".internal.alerts-observability.logs.alerts-default-000001",
        ".ds-metrics-elastic_agent.osquerybeat-default-2024.07.21-000001",
        ".ds-metrics-system.socket_summary-default-2024.07.21-000001",
        ".kibana-observability-ai-assistant-kb-000001",
        ".ds-logs-system.auth-default-2024.07.21-000001",
        ".ds-metrics-system.diskio-default-2024.07.21-000001",
        ".ds-metrics-system.network-default-2024.07.21-000001",
        ".ds-.kibana-event-log-ds-2024.07.21-000001",
        ".internal.alerts-ml.anomaly-detection.alerts-default-000001",
        ".apm-custom-link",
        ".ds-logs-elastic_agent.metricbeat-default-2024.07.21-000001",
        ".logs-osquery_manager.actions-default",
        ".internal.alerts-observability.threshold.alerts-default-000001",
        ".ds-metrics-elastic_agent.filebeat-default-2024.07.21-000001",
        ".transform-notifications-000002",
        ".ds-.slm-history-7-2024.07.21-000001",
        ".kibana_analytics_8.14.3_001",
        ".kibana_8.14.3_001",
        ".security-7",
        ".slo-observability.summary-v3.2.temp",
        ".ds-metrics-system.uptime-default-2024.07.21-000001",
        ".logs-osquery_manager.action.responses-default",
        ".slo-observability.summary-v3.2",
        ".ds-.logs-deprecation.elasticsearch-default-2024.07.21-000001",
        ".ds-logs-elastic_agent-default-2024.07.21-000001",
        ".security-tokens-7",
        ".geoip_databases",
        ".ds-logs-elastic_agent.filebeat-default-2024.07.21-000001",
        ".internal.alerts-transform.health.alerts-default-000001",
        ".ds-.fleet-actions-results-2024.07.21-000001",
        ".internal.alerts-default.alerts-default-000001",
        ".ds-metrics-system.load-default-2024.07.21-000001",
        ".fleet-actions-7",
        ".ds-metrics-system.process.summary-default-2024.07.21-000001",
        ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001",
        ".security-profile-8",
        ".fleet-enrollment-api-keys-7",
        ".ds-logs-system.syslog-default-2024.07.21-000001",
        ".ds-metrics-system.fsstat-default-2024.07.21-000001",
        ".kibana_alerting_cases_8.14.3_001",
        ".fleet-policies-leader-7",
        ".apm-agent-configuration",
        ".ds-metrics-fleet_server.agent_versions-default-2024.07.21-000001",
        ".fleet-policies-7",
        ".kibana_security_session_1",
        ".internal.alerts-observability.slo.alerts-default-000001",
        ".ds-logs-elastic_agent.osquerybeat-default-2024.07.21-000001",
        ".ds-metrics-system.memory-default-2024.07.21-000001",
        ".ds-metrics-system.filesystem-default-2024.07.21-000001",
        ".internal.alerts-security.alerts-default-000001",
        ".ds-metrics-elastic_agent.elastic_agent-default-2024.07.21-000001",
        ".kibana-observability-ai-assistant-conversations-000001",
        ".internal.alerts-observability.uptime.alerts-default-000001",
        ".ds-metrics-system.cpu-default-2024.07.21-000001"
      ],
      "data_streams": []
    }
  ]
}

Observe that the data_streams object is empty.

With verbose = true:

{
  "total": 1,
  "remaining": 0,
  "snapshots": [
    {
      "include_global_state": true,
      "uuid": "PSIEWRmOR1m5EzTiCZJANg",
      "repository": "found-snapshots",
      "duration_in_millis": 8821,
      "start_time": "2024-07-22T02:29:59.815Z",
      "shards": {
        "successful": 71,
        "failed": 0,
        "total": 71
      },
      "version_id": 8505000,
      "end_time_in_millis": 1721615408636,
      "state": "SUCCESS",
      "version": "8.14.0-8.14.2",
      "snapshot": "cloud-snapshot-2024.07.22-wpnxv4hmqzqb3vewczcjvq",
      "end_time": "2024-07-22T02:30:08.636Z",
      "feature_states": [
        {
          "indices": [
            ".security-tokens-7",
            ".security-7",
            ".security-profile-8"
          ],
          "feature_name": "security"
        },
        {
          "indices": [
            ".kibana_8.14.3_001",
            ".kibana_security_solution_8.14.3_001",
            ".apm-custom-link",
            ".kibana_ingest_8.14.3_001",
            ".apm-agent-configuration",
            ".kibana_analytics_8.14.3_001",
            ".kibana_security_session_1",
            ".kibana_alerting_cases_8.14.3_001",
            ".kibana_task_manager_8.14.3_001"
          ],
          "feature_name": "kibana"
        },
        {
          "indices": [
            ".geoip_databases"
          ],
          "feature_name": "geoip"
        },
        {
          "indices": [
            ".transform-internal-007"
          ],
          "feature_name": "transform"
        },
        {
          "indices": [
            ".fleet-agents-7",
            ".fleet-enrollment-api-keys-7",
            ".fleet-actions-7",
            ".fleet-policies-7",
            ".fleet-servers-7",
            ".fleet-policies-leader-7"
          ],
          "feature_name": "fleet"
        }
      ],
      "indices": [
        ".ds-metrics-system.memory-default-2024.07.21-000001",
        ".apm-agent-configuration",
        ".ds-metrics-system.socket_summary-default-2024.07.21-000001",
        ".ds-metrics-elastic_agent.metricbeat-default-2024.07.21-000001",
        ".ds-logs-osquery_manager.result-default-2024.07.21-000001",
        ".kibana_task_manager_8.14.3_001",
        ".fleet-servers-7",
        ".kibana_ingest_8.14.3_001",
        ".ds-ilm-history-7-2024.07.21-000001",
        ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001",
        ".slo-observability.summary-v3.2",
        ".ds-metrics-elastic_agent.elastic_agent-default-2024.07.21-000001",
        ".internal.alerts-observability.metrics.alerts-default-000001",
        ".internal.alerts-ml.anomaly-detection.alerts-default-000001",
        ".internal.alerts-security.alerts-default-000001",
        ".apm-source-map",
        ".logs-osquery_manager.action.responses-default",
        ".kibana_security_solution_8.14.3_001",
        ".ds-.kibana-event-log-ds-2024.07.21-000001",
        ".ds-metrics-fleet_server.agent_status-default-2024.07.21-000001",
        ".fleet-actions-7",
        ".fleet-enrollment-api-keys-7",
        ".internal.alerts-observability.apm.alerts-default-000001",
        ".ds-metrics-fleet_server.agent_versions-default-2024.07.21-000001",
        ".ds-logs-elastic_agent.metricbeat-default-2024.07.21-000001",
        ".kibana-observability-ai-assistant-conversations-000001",
        ".internal.alerts-observability.threshold.alerts-default-000001",
        ".kibana_alerting_cases_8.14.3_001",
        ".fleet-policies-leader-7",
        ".ds-metrics-system.network-default-2024.07.21-000001",
        ".ds-logs-system.syslog-default-2024.07.21-000001",
        ".ds-.slm-history-7-2024.07.21-000001",
        ".logs-osquery_manager.actions-default",
        ".geoip_databases",
        ".ds-metrics-system.load-default-2024.07.21-000001",
        ".ds-.logs-deprecation.elasticsearch-default-2024.07.21-000001",
        ".internal.alerts-observability.slo.alerts-default-000001",
        ".ds-metrics-system.cpu-default-2024.07.21-000001",
        ".internal.alerts-observability.uptime.alerts-default-000001",
        ".fleet-agents-7",
        ".security-7",
        ".ds-metrics-elastic_agent.osquerybeat-default-2024.07.21-000001",
        ".transform-notifications-000002",
        ".kibana_security_session_1",
        ".internal.alerts-stack.alerts-default-000001",
        ".internal.alerts-default.alerts-default-000001",
        ".ds-metrics-elastic_agent.filebeat-default-2024.07.21-000001",
        ".slo-observability.summary-v3.2.temp",
        ".ds-metrics-system.uptime-default-2024.07.21-000001",
        ".security-tokens-7",
        ".ds-metrics-elastic_agent.filebeat_input-default-2024.07.21-000001",
        ".slo-observability.sli-v3.2",
        ".ds-logs-elastic_agent.filebeat-default-2024.07.21-000001",
        ".fleet-policies-7",
        ".apm-custom-link",
        ".kibana_analytics_8.14.3_001",
        ".internal.alerts-transform.health.alerts-default-000001",
        ".ds-logs-elastic_agent.osquerybeat-default-2024.07.21-000001",
        ".ds-logs-system.auth-default-2024.07.21-000001",
        ".ds-metrics-system.process-default-2024.07.21-000001",
        ".ds-metrics-system.process.summary-default-2024.07.21-000001",
        ".ds-metrics-system.fsstat-default-2024.07.21-000001",
        ".security-profile-8",
        ".ds-logs-elastic_agent-default-2024.07.21-000001",
        ".internal.alerts-observability.logs.alerts-default-000001",
        ".ds-metrics-system.diskio-default-2024.07.21-000001",
        ".ds-metrics-system.filesystem-default-2024.07.21-000001",
        ".transform-internal-007",
        ".ds-.fleet-actions-results-2024.07.21-000001",
        ".kibana-observability-ai-assistant-kb-000001",
        ".kibana_8.14.3_001"
      ],
      "failures": [],
      "data_streams": [
        ".fleet-actions-results",
        ".logs-deprecation.elasticsearch-default",
        "ilm-history-7",
        "logs-elastic_agent.osquerybeat-default",
        "metrics-system.process-default",
        "metrics-elastic_agent.filebeat-default",
        "metrics-elastic_agent.metricbeat-default",
        "metrics-elastic_agent.filebeat_input-default",
        "metrics-system.process.summary-default",
        "metrics-system.network-default",
        "logs-system.auth-default",
        "metrics-system.load-default",
        "logs-elastic_agent.metricbeat-default",
        "logs-osquery_manager.result-default",
        "metrics-system.fsstat-default",
        "logs-system.syslog-default",
        "metrics-system.cpu-default",
        "logs-elastic_agent.filebeat-default",
        "metrics-system.memory-default",
        "metrics-elastic_agent.elastic_agent-default",
        "logs-elastic_agent-default",
        "metrics-elastic_agent.osquerybeat-default",
        ".kibana-event-log-ds",
        ".slm-history-7",
        "metrics-system.diskio-default",
        "metrics-system.filesystem-default",
        "metrics-fleet_server.agent_status-default",
        "metrics-system.uptime-default",
        "metrics-fleet_server.agent_versions-default",
        "metrics-system.socket_summary-default",
        ".fleet-actions-results"
      ],
      "start_time_in_millis": 1721615399815,
      "metadata": {
        "policy": "cloud-snapshot-policy"
      }
    }
  ]
}

Logs (if relevant)

No response

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner commented 1 month ago

This is true, but in fact the clients team have indicated a preference for maintaining the structure of responses no matter what, rather than having fields which appear or disappear depending on context or response content. So I think the fix here would be to prefer to output all the fields in the response even when empty.