search

tableau / extensions-api

Extensions API sample code and developer docs.
http://tableau.github.io/extensions-api
MIT License
268 stars 251 forks source link

Memory limit in Extensions API? #357

Closed merlijnbuit closed 2 years ago

merlijnbuit commented 3 years ago

Hi all,

We received an issue from a customer which got a 'no data error' in SuperTables on Tableau Desktop after he added more data.

I have also tested this and I get the error when I go above 150.000 rows with my demo dataset. It looks like Tableau reaches a memory limit or some timeout. There is no error in my debug console so I can't figure out why it crashes.

Please see attached video: https://youtu.be/regKxoNoXRU

Tested on both 2020.2.4 and 2020.3.2, Mac and Windows.

This issue does not appear on Tableau Server. Tested on version 2020.3.0 (Tableau Online)

Merlijn

johnDance commented 3 years ago

Hello Merlijn Thanks for your report and narrowing down the scope of the problem. We will investigate and get back with you. John

merlijnbuit commented 3 years ago

Hello John,

Thank you appreciate it.

merlijnbuit commented 3 years ago

Hello John,

I have also emailed this directly to you and your team a few weeks ago. Is there any progress to report?

Issues like this are killing the user experience with extensions. We really need a stable solution in order to extend the Tableau Platform, some of your biggest clients rely on extensions and are having these issues.

An update is greatly appreciated.

johnDance commented 3 years ago

Hi Merlijn This report is at the top of our defect list. We had a slight delay digging into it because of another issue that came up. But we are looking into it this sprint. John

demartsc commented 3 years ago

This could be the same issue affecting this user of the kepler.gl extension as well. https://github.com/keplergl/kepler.gl-tableau/issues/35. I have noticed in testing larger data sets with Kepler if I go beyond 250k (with small number of columns) I can run into this scenario.

johnDance commented 3 years ago

Hi @demartsc We have identified the issue and are working to reduce payload size and improve performance for getSummaryDataAsyc. There is a memory size constraint in the communication between tableau desktop and the extension. We are restructuring the payload to greatly reduce the size. We have also improved the speed of getSummaryDataAsync. In addition to the reduced payload size, we will add some parameters that extension developers can use to further reduce payload size and improve performance. The options will be: onlyNativeValues/onlyFormattedValues. You can pass these to the command if you only look at DataValue.value/nativeValue or only look at DataValue.formattedValue. I'm guessing kepler.gl just looks at value/nativeValue fields. We will also add includeColumns to specify the columns that you really want, and getSummaryDataColumnsAsync, to just get the column meta data. We hope to get all these in for 2021.2. While these will really help, there will still be a number of rows that will strain performance and exceed memory constraints. Soon after 2021.2 we plan to implement a paging/streaming approach to getSummaryDataAsync, which should completely solve all the issues. Let me know if you have any questions, or contact @Kovner for more details. Thanks, John