jupyterlab / jupyterlab-data-explorer

First class datasets in JupyterLab
BSD 3-Clause "New" or "Revised" License
178 stars 38 forks source link

Notes from meeting with Tim #103

Closed saulshanabrook closed 1 year ago

saulshanabrook commented 5 years ago

I met with @timkpaine today to see they are up to at JP Morgan in exposing SQL databases and virtual filesystems in JupyterLab.

Currently, they are using something like this multi contents manager (I can't remember if they are using that exactly or an internal fork of it). My impression from it is that the Jupyter server supports only one contents manager. So this project lets you combine multiple contents managers into one contents manager. You still specify a default that shows up normally, so the filebrowser looks the same in JL. But, you add some special root paths, like /s3:// to redirect to other content managers under the hood.

Then in JL, using https://github.com/youngthejames/jupyterlab_filetree/, they display these other roots in different browsers in JL. They wanna use this to make a "SQL" contents manager that shows different tables as files.

My takeaways was that building a good tree view UI is a PITA and so the data registry should be the place we do that and then others don't have to re-implement it. Tim pointed out a bunch of subtle behavior that they copied from the Finder on mac in terms of single and double clicking on files and folders. Also it would be nice to have both a tree view and a multipanel view the user could switch between.

We should support the contents manager API in the data registry, with the end goal of replacing JL's existing file browser with the data browser. A bunch of things need to happen before that's feasible including:

I also got a nice demo of perspective. In our world, it functions a bit like voyager or plotly. However, it is much more performant for large datasets. So we discussed how it could use the data registry mimetypes to represent different pointers to different types of data like parquet, and use that.

vidartf commented 5 years ago

Might it be worthwhile to look into making a generic tree-widget, that is agnostic about what it is showing? If we are having multiple such tree views, we might save some effort?

quigleyj-mavenomics commented 5 years ago

If you're interested, I've written a tree control to address some of these use cases that's being used internally. We're about to open-source a larger dashboarding project that encompasses it, but I can separate out the tree control and open-source that sooner.

There's some rough edges that need to be cleaned up, but it was designed to address a lot of the shortcomings of other tree controls (inflexibility, no async support, bad drag UX, no tree-to-tree, etc). It's written in React, but wraps nicely in a Phosphor widget.

Here it is in action, with a backend making updates to the tree asynchronously (in this case, doing some tree pruning to remove useless containers) tree-control-demo

saulshanabrook commented 5 years ago

@quigleyj-mavenomics Oh man that would be perfect! If you do release it, I will definitely try integrating it here. It would be great to share this as a nice reusable react component.

jtpio commented 5 years ago

@saulshanabrook I remember seeing somewhere that you were considering reusing the tree viewer from VS Code. Any chance making it work in JupyterLab?

edit: mentioned in https://github.com/jupyterlab/jupyterlab-data-explorer/issues/17

saulshanabrook commented 5 years ago

I started looking into it but found it hard to separate from vscodes build system.

That would be nice though! Maybe I'll have another go at it.

timkpaine commented 4 years ago

FYI starting to work on some of the perspective stuff for data https://github.com/finos/perspective/pull/824

telamonian commented 4 years ago

@quigleyj-mavenomics What's the status of React tree component? Have you open sourced it anywhere?

For context, I'm starting to work on a refactor of the filebrowser UI and underlying stack in jlab core. One of my goals is to make the file listing a tree view. I'm deciding right now whether to use an existing React component for the tree, or if I should roll a new one. And then I remembered this thread, and thought I'd check in with you