Enhance the data viewer

[renkun-ken]

Currently, we are using DataTables in the WebView as the data viewer with session watcher enabled. It has the following capabilities (but we are not using all of them at the moment):

• Sorting a column
• Always show column header on page top
• Different background color of odd/even rows
• Highlight row on hover
• Highlight rows on click
• Search text and filter rows over the entire table
• Paging (not used yet)

There are some alternative data viewer widgets that could provide more power features such as column filtering by value or range (like in RStudio), multi-column ordering. I guess we could investigate into this and see if we could provide a more powerful data viewer.

More powerful alternatives:

• Handsontable (maybe there's potential license issue)

[alex-gable]

@renkun-ken if the viewer could honor light+dark or the vscode theme it would be incredible. it's a bit blinding cracking it open at night

[Ikuyadeu]

As the related work, vscode-jupyter implements similar function by the react and tsx. Imitating it is cost, but safe for the license issue.

• variableViewPanel.tsx
• variablePanel.tsx

[gvelasq]

@Ikuyadeu I support the idea of porting the vscode-jupyter data viewer. I filed a related issue in vscode-jupyter: https://github.com/microsoft/vscode-jupyter/issues/4604

When I run View(mtcars) in a Jupyter notebook in VS Code Insiders, I get the following error:

ERROR: Error in View(mtcars): 'View()' not yet supported in the Jupyter R kernel

My hope is that the 'not yet' in the error implies that adding R support for the data viewer could already be a priority for vscode-jupyter, in which case porting to vscode-R may be significantly easier if R data structures have already been specified.

For anyone who has not yet done so, I would encourage you to upvote R here: https://github.com/microsoft/vscode-jupyter/issues/1536

[gvelasq]

To follow up on the issue I raised in vscode-jupyter, this very helpful response from a Microsoft employee points to two files that might be a useful starting point: https://github.com/microsoft/vscode-jupyter/issues/4604#issuecomment-772043821

[renkun-ken]

The data viewer of SQL Tools looks good too. The source seems to be at https://github.com/mtxr/vscode-sqltools/blob/master/packages/vscode/webview-provider/index.ts.

From https://github.com/mtxr/vscode-sqltools/blob/master/packages/plugins/package.json, it uses DevExpress/devextreme-reactive.

[RandomFractals]

@renkun-ken you might want to try canvas based grids:

https://github.com/TonyGermaneri/canvas-datagrid

https://github.com/fin-hypergrid/core

Handsontable looks really good, but I am not sure their licensing is suitable for vscode extensions needs.

[danielbasso]

I'm using this extension for about one month now and one of the main issues I have is that the current dataframe view methods (using View() or R: Preview Dataframe) is that the have to load all data. As dataframes get bigger these methods start to become slow, unresponsive or even can't be used at all.

One of the nicest features of Rstudio dataframe preview (I know these comparisons to Rstudio are boring, but they are the 'gold standard' for R programming) is that it dynamically loads about 20-30 rows at a time and the first 50 columns; this is nice, because no matter how big your data is, you can have a preview of it and apply some simple filters.

Between View() or R: Preview Dataframe I prefer the latter, because it's prettier, integrates better with the global theme and has better filtering options:

View() (kind of ugly, few filtering options):

R: Preview Dataframe (way better looking - similar to RStudio - better filters, but has to load all data and limited to 20 Mb):

For me, the R: Preview Dataframe as it is with dynamically loading would be near perfect. As a workaround, right now I do things like previewing my_df[1:100, ] but it's not ideal.

[danielbasso]

@gvelasq @renkun-ken @Ikuyadeu I've been experimenting with the jupyter api for the last two days. Results so far:

What do you guys think? Should I proceed? Bad idea? The downside is it requires that the jupyter extension is installed; but my idea would be to default to the current viewer if the extension isn't present.

What I liked so far:

• Has a dynamic loading implemented; as far is I tested, it will automatically request batches of rows based on what the user is currently seeing (allow arbitrary large data frames to be viewed);
• Asynchronous, so it won't lock the R terminal, and even if it hangs/becomes unresponsive, the rest of vscode usually proceeds without harm;
• The data frame has to be mapped through an instance of a class, OOP makes things smoother to implement and maintain;
• Is very fast for "normal" needs, loads almost without delay for a data frame of 100000 rows and 12 columns; for an excessive big data frame, without the dynamic loading, it loaded in about 20 seconds;
• It's an API, so basically we wouldn't need to maintain it's code.

Downsides:

• Requires jupyter extension installed (workaround: default to another viewer method if the extension isn't present);
• It's an API, so we don't have full control of it's capabilities.

I read the Webview API documentation and unfortunately it seems not very well suited for interactivity (dynamic paging, for example). It is technically possible, but I'm not sure it's the best direction we could go.

[Ikuyadeu]

@danielbasso Thank you for the nice first try. Depending on Jupyter extension is no problem. We are considering connecting some functions to that. We can move these functions to Jupyter extension after it is stabled in this extensions.

[gvelasq]

@danielbasso thanks very much for taking a look at the vscode-jupyter API. The screenshot looks like a great start.

[renkun-ken]

It looks like ag-grid could be used as a better data viewer if we don't rely on external viewers. It has a ag-grid-community distributed under MIT license and is free to use in any production system.

[QizhiSu]

Hello guys, I am wondering if it is possible to show column index when hovering on the header and showing the dimension of the dataframe. I think it can be very useful. By the way, automatic updating of the dataframe if it has been changed is also nice.