timeout on enhanced table visualization

[allardwp]

On our production environment we are running ELK 7.6.2. with some Enhanced Table visualizations without any problems.

On our acceptance environment with the exact same data we have upgraded ELK to 7.9.3 and also upgraded the Enhanced Table plugin. However we are now getting timeouts on the dashboard and visualization. First a message appears "Your query is taking a while", and then an "Aborted" message appears.

We are using quite a simple configuration for the Enhanced Table with:

• split on rows
• spit on cols
• sum of specific field

I suspect that it has to do with a new way that Kibana is retrieving data for its visualizations. In v7.6.2 Kibana does a call to "_search" for each visualization. But in v.7.9.3 I can see in the debugger that each second there are many requests to "/es" where the "took" property is increasing, untill it finally times out. If I execute the query directly in ES, then it returns in <50 ms.

One of the responses looks like this :{"id":"FjdMYU...jE=","is_partial":true,"is_running":true,"rawResponse":{"took":4678,"timed_out":false,"terminated_early":false,"num_reduce_phases":0,"_shards":{"total":1,"successful":0,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}},"total":1,"loaded":0}

All other visualizations of Kibana are running without problems. It is only the Enhanced Datatable that seems to cause problems. Is there some problem in the plugin in combination with this new Kibana version ?

[fbaligand]

Well, the major change is that, starting from Kibana 7.9, Enhanced Table plugin has been migrated to Kibana new platform. It brings some new features like "Drilldowns" or Canvas compatibility. But I don't see any major change concerning the way data is requested and loaded. That's said, since Kibana 7.8, data loading in Dashboards has globally changed in Kibana: each visualization does an individual query, whereas in previous versions, only one "msearch" query is done by Kibana for all visualizations.

That said, I did specific tests with Kibana 7.9 and didn't get timeouts, even with 1000 rows. Do you have "auto-refresh" activated on your dashboard and timeout happens after multiple refreshs?

[allardwp]

thanks for your quick response. I do not have auto-refresh activated.

Some additional information I can add: the problem does not seem to happen always. Sometimes the visualization loads almost immediately, but when changing the date filter the timeout sometimes appears. Sometimes after refreshing the same page (with same filter) it is appearing at once again.

I think the problem is related to the number of buckets and maybe paging ? ... : my resulting table has 23 columns and many rows (pages of 10 rows per page). All this information is relevant to the users (they need to be able to page through all results). If I lower the number of rows then the problem disappears.

For example with a maximum of 5 rows, I see only 2 /es queries and the querying stops. With a maximum number of 100 rows, I already see 16 queries and the querying stops. So when specifying a higher maximum number of rows I see more /es queries of Kibana, and the timeout happens.

Altering the number of rows per page has no effect.

Again, with exactly the same data on version 7.6.2 the problem does not happen, and when executing the query directly to Elastic Seach the results are coming back fast. So I guess the problem is not related to the query to Elastic Search, but somewhere in Kibana ?

Does this help to pinpoint the problem in 7.9.3 ?

[fbaligand]

Yes, it helps. You have to know that globally, whatever your pagination, every row is loaded in memory. Pagination is just rendering.

That said, I'm very surprised that you see several ES queries. Normally,whatever the number of rows, there should have only one ES query.

Can you share the different buckets/metrics configured?

[allardwp]

I think I found it ! I configured the row split on a scripted field. (that is the concatenation of a customer name and the customer id). So each row I am presenting in the table is grouped by customer, and horizontally there are product categories. I am showing the total sale for each customer/category.

If I change "split rows" to the customer.name.keyword field then the problem does not happen anymore. This seems to be consistent.

So I guess for now there is a workaround. Still I would like to use the scripted field if that would be possible. I know a way around this could be to actually store that in a separate field.

[allardwp]

Well, I screamed with joy too soon .. seems that in Visualization mode the problem indeed does not happen anymore, but in Dashboard mode it still happens. Even when it is the only Visualization on the dashboard.

[fbaligand]

So:

• you have only one "split rows" bucket? How many rows to get the issue?
• you reproduce the issue only on dashboard view, not on visualize view?
• with dashboard view and only enhanced table vis, you see multiple elasticsearch queries called?

Could you:

• on dashboard view, go to "inspect" menu of enhanced table vis, check request stats (request time, elapsed time), copy the request and execute it on dev tools?
• try similar config with core data table? (Similar buckets & metrics) And tell me if you reproduce the same issue or not.

[allardwp]

My answers:

• you have only one "split rows" bucket? Correct How many rows to get the issue? Not consistent, sometimes with 9 rows I get already one additional es query (in visualize view)
• you reproduce the issue only on dashboard view, not on visualize view? it happens on both, but seems to happen MUCH more on dashboard view than on visualize view, even when I select 9 rows
• with dashboard view and only enhanced table vis, you see multiple elasticsearch queries called? correct

See here the request stats:

Hits | 0 Hits (total) | 388352 Query time | 535ms Request timestamp | 2020-11-12T08:04:10.491Z

(so probably this hits (total) is suspicious, right ?)

When executing the request on dev tools it executes normally. ("took" : 477)

I tried similar config with core data table (without splitting on cols), and there is no issue there.

[allardwp]

See here an image to clarify

[allardwp]

hmm now it seems that is was related to the option : Group other values in separate bucket This was enabled on the split rows bucket. When this is enabled, the problem seems to occur. When I disable it, the problem seems to have gone. Since I am already getting all the buckets (about 1000 in total), there is no need for this option. For now this seems to have resolved it both on visualization view and dashboard view.

[fbaligand]

Nice to see that you fixed your issue!