search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.68k stars 8.23k forks source link

[Security Solution][DataQuality Dashboard][Serverless] Fix stats api filtering logic #196528

Closed kapral18 closed 5 days ago

kapral18 commented 1 month ago

Describe the bug:

Currently serverless kibana DQD /internal/ecs_data_quality_dashboard/stats/{pattern} api logic fetches only indices that have data in a specific range (1 week) https://github.com/elastic/kibana/blob/47a497cf3671eee56adeb4787cb4c9271671f5e8/x-pack/plugins/ecs_data_quality_dashboard/server/helpers/get_available_indices.ts#L34

This effectively filters out any indices that have docs without @timestamp fields or don't have any docs yet.

This is a problem because ess implementation doesn't do that and shows missing @timestamp field in the latest check report both in incompatible fields and as part of missing ecs compliant fields ui.

To align these implementations we need to use index creation data api from es (example GET /_cat/indices/.alerts-security.alerts-default?h=creation.date) instead of relying on filtering docs by @timestamps in serverless.

This results in empty pattern tables in serverless for empty indices or indices without @timestamp, which is confusing, and when of empty indices are stacked it pushes out the non-empty ones out of the view.

This can be circumvented with a temporary fix to close them by default or sort them by content, but it's arguably easier to just fix the core issue altogether.

Kibana/Elasticsearch Stack version: all supported versions

Steps to reproduce:

  1. Open Kibana Serverless DevTools
  2. Add empty index
  3. Set data views including that index in a pattern
  4. Check dataquality dashboard and see pattern present but index missing

Current behavior:

Pattern is showing but empty

Expected behavior: Pattern is showing latest index to check

Screenshots (if relevant): Image

elasticmachine commented 1 month ago

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine commented 1 month ago

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

MadameSheema commented 5 days ago

Is this the new expected behavior?

Image

Image

If so, please feel free to close the ticket.

Thanks!! :)

kapral18 commented 5 days ago

@MadameSheema yes our currently fixed implementation shows incompatible field mappings even if no data is present for mappings that are either created within a week or are created earlier but have received data within a week