Kibana Lens HeatMap unexpected/unclear behavior for the "include empty rows" flag for Date Histograms

[herrBez]

Kibana version: 8.10.1

Elasticsearch version: 8.10.1

Server OS version: N/A

Browser version: Chrome Version 117.0.5938.89 (Official Build) (64-bit)

Browser OS version: Windows 11

Original install method (e.g. download page, yum, from source, etc.): Elastic Cloud

Describe the bug: Inconsistent behavior for the Flag "Include empty rows" in Kibana Lens Histograms.

Steps to reproduce:

1. Create an Index for the data

   PUT metrics-service-test/_bulk
   {"create":{}}
   {"@timestamp":"2023-09-20T00:00:00.000Z","service.name":"apmserver"}
   {"create":{}}
   {"@timestamp":"2023-09-20T00:00:00.000Z","service.name":"apmserver1"}

2. Create a Kibana Lens Heat Map:

   • Select Date Histogram on @timestamp field (deactivate the flag "include empty rows")
   • Select Top values service.name as vertical Axis
   • Select Count Records as Cell Value

{"attributes":{"description":"","state":{"adHocDataViews":{"efd875fb-8719-43fe-8c33-d885986c7e63":{"allowNoIndex":false,"fieldAttrs":{},"fieldFormats":{},"id":"efd875fb-8719-43fe-8c33-d885986c7e63","name":"metrics-service-test","runtimeFieldMap":{},"sourceFilters":[],"timeFieldName":"@timestamp","title":"metrics-service-test"}},"datasourceStates":{"formBased":{"layers":{"3ef551b3-3dc3-433f-8cdf-e2e4cddd587c":{"columnOrder":["add67950-c283-4521-b4f4-6527f9ab44ec","733d965b-401f-4841-9f1f-61ec16041484","2ca1d221-1526-4395-9b79-684856900c27"],"columns":{"2ca1d221-1526-4395-9b79-684856900c27":{"dataType":"number","isBucketed":false,"label":"Count of records","operationType":"count","params":{"emptyAsNull":true},"scale":"ratio","sourceField":"___records___"},"733d965b-401f-4841-9f1f-61ec16041484":{"dataType":"date","isBucketed":true,"label":"@timestamp","operationType":"date_histogram","params":{"dropPartials":false,"includeEmptyRows":true,"interval":"auto"},"scale":"interval","sourceField":"@timestamp"},"add67950-c283-4521-b4f4-6527f9ab44ec":{"dataType":"string","isBucketed":true,"label":"Top 3 values of service.name","operationType":"terms","params":{"exclude":[],"excludeIsRegex":false,"include":[],"includeIsRegex":false,"missingBucket":false,"orderBy":{"columnId":"2ca1d221-1526-4395-9b79-684856900c27","type":"column"},"orderDirection":"desc","otherBucket":true,"parentFormat":{"id":"terms"},"size":3},"scale":"ordinal","sourceField":"service.name"}},"ignoreGlobalFilters":false,"incompleteColumns":{},"sampling":1}}},"indexpattern":{"layers":{}},"textBased":{"layers":{}}},"filters":[],"internalReferences":[{"id":"efd875fb-8719-43fe-8c33-d885986c7e63","name":"indexpattern-datasource-layer-3ef551b3-3dc3-433f-8cdf-e2e4cddd587c","type":"index-pattern"}],"query":{"language":"kuery","query":""},"visualization":{"gridConfig":{"isCellLabelVisible":false,"isXAxisLabelVisible":true,"isXAxisTitleVisible":false,"isYAxisLabelVisible":true,"isYAxisTitleVisible":false,"type":"heatmap_grid"},"layerId":"3ef551b3-3dc3-433f-8cdf-e2e4cddd587c","layerType":"data","legend":{"isVisible":true,"position":"right","type":"heatmap_legend"},"shape":"heatmap","valueAccessor":"2ca1d221-1526-4395-9b79-684856900c27","xAccessor":"733d965b-401f-4841-9f1f-61ec16041484","yAccessor":"add67950-c283-4521-b4f4-6527f9ab44ec"}},"title":"KibanaBug","visualizationType":"lnsHeatmap"},"coreMigrationVersion":"8.8.0","created_at":"2023-09-20T13:48:52.118Z","id":"6ee73b60-57bc-11ee-b21b-353481f74d1f","managed":false,"references":[],"type":"lens","typeMigrationVersion":"8.9.0","updated_at":"2023-09-20T13:48:52.118Z","version":"WzEwMTA2LDIzXQ=="}
{"excludedObjects":[],"excludedObjectsCount":0,"exportedCount":1,"missingRefCount":0,"missingReferences":[]}

3. Select a Timerange that include the datasets (e.g., last 30 days)

4. Verify that now one entry for each Date Interval is shown

5. Filter for service.name: apmserver ==> Verify that only one unique time bucket is shown/

Expected behavior: It is not clear whether initially we should only see a unique date bucket or not (given that only one time interval contains data to show)

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[stratoula]

This is indeed a confusing behavior. We discussed it with the team and we think that in the case of only one bucket and Include empty rows switch off the chart would be better to behave as it behaves when the filter is applied.