Complex dataset in debugging pane causes WebView2 error

[PBI-David]

I'm using the latest 1.6 build (ALPHAdeneb7E15AEF80B9E4D4F8E12924291ECE89A.1.6.0.20230905.71bd38a)

If you open my calendar heat map example here (https://github.com/PBI-David/Deneb-Showcase/tree/main/Calendar%20Heatmap). In the old 1.5.1 version, I can successfully navigate to the cell table below:

Doing the same in 1.6 causes a very bad crash of PBI.

Incidentally, did you move the table dropdown to the bottom to save space? Have you considered putting the pagination and table dropdown at the top to be consistent with the Vega editor? I'll get used to it either way.

[dm-p]

Thanks for finding!

Root cause seems to be that pruning and parsing the cell dataset results in a 322 MB object that is hitting a memory limit when we attempt to restructure it, and this is killing the WebView2 thread. In versions prior to 1.6 we used our own function to manage this, and we've moved to using the same one that Vega Editor does in 1.6 (as this solved some problems we were having with other datasets). I'll have a look as to what we can do here to mitigate but might be a tricky one (although it will need to be fixed prior to moving to beta).

Regarding the dropdown list, yes, this is for space saving purposes - it's mentioned in the change log. It's also done for consistency with the editor pane, where the status bar for the JSON editor is placed (and this will change in 1.8 to be a custom component rather than JSONEditor's one). We can probably make this configurable if you prefer the debug area status bar to be at the top, so feel free to create an issue for it if it's something you need.

[dm-p]

Further thoughts:

• Working hypothesis is that the function we're using is fine; it's how we're processing the dataset that is problematic.
• <= 1.5 processes the data at a cell level for display, which is more manageable.
• In 1.6 we are attempting to pre-process the data before it hits the new worker, which is too much overhead for the main thread.
• We avoid using the data table worker for this because we can't reference modules within the custom visual (as they cannot be lazy loaded).
• We should ideally be able to pass our method that we use for sanitizing and cleaning up complex objects to the worker (or clone it in there) so that we can do this in the worker thread.
• if we can do this successfully, we can eke some more performance out of the main thread when we change datasets and delegate the whole thing off to the worker, which will keep the UI responsive until the data is computed at a cell level.

EDIT:

By moving the pruning of complex objects to the cell level, this causes issues with hashing the dataset, to determine if it has changed prior to sending to the data worker, so we will still need something at this stage to remove deep and circular references. We will need to look at the performance of our current approach and make it better.

[dm-p]

From further analysis, this is a special case where Vega-Lite creates a Vega dataset that includes the scenegraph (which is an enormous object of its own). Attempting to prune this is expensive and killing the main thread.

As this would end up being pruned eventually anyway, I have modified the pruning function to hard cut an object property that contains the Vega scenegraph so that we don't have to waste further resources processing it.

This results in a much faster operation for viewing the dataset and Deneb should be much more stable, e.g.:

https://github.com/deneb-viz/deneb/assets/10572054/50d321ac-2bb1-436f-9904-59f3449aaad0

There is a new alpha build available for download, and if you're able to re-test this scenario, plus some of your more complex visuals to confirm that there's no further gremlins, that would be fantastic :)

(and thanks once again for finding this one)

[PBI-David]

Confirmed as fixed. Smooth as butter.

[dm-p]

Excellent - thanks very much for confirming!