Pivottable as an IPyWidget?

psychemedia commented 8 years ago

I'm not sure how this would work, but I wonder if as well as using the pivot table to explore and interact with data from a pandas dataframe, it might be useful to provide a mechanism for passing the data back from the pivottable to a pandas dataframe using an IPyWidget mechanic? (Some related ideas in the context of dashboards here.)

For teaching purposes, it might also be handy to provide a view of pandas code that mimics any operations performed in the pivottable? (If there was no return path of data from the widget back to pandas, at least the user could copy the generated code and apply it to the original dataframe using pandas?

bollwyvl commented 8 years ago

ref https://github.com/jupyter/jupyter/issues/124

I agree wholeheartedly. Enough to have done the bulk of the work!

Here is an example of widgetizing all the low-hanging fruit.

I started clean slate from the cookiecutter on a new repo, so I can't PR without some work... and it may be appropriate to offer it as a second option, anyway, rather than changing this repo. At the least, we could still provide the base API function from the current repo.
I passed on the special pandas integration in the widget: there is no canonical approach yet, i.e. traitlets.DataFrame, and it would remain to be seen how deep you'd want it... of course, the whole df.pivot API would be awesome!
- Right now, I am using onRefresh to update the stuff that a user can fiddle with (except for inclusions/exclusions). Is there a way to get access to the post-pivot data? That would certainly be a start for getting it back!
iframes make me lose sleep at night. It would be very nice to be able to reach down and replace all the references to window.width!

Hope this helps!

nicolaskruchten commented 8 years ago

This is amazing, thank you! I'll definitely dig into this over the next couple of days in between diaper changes (I'm on parental leave so my open-source availability is very spikey!) and I will likely just squash my repo with something like this, which has many technical advantages :)

bollwyvl commented 8 years ago

Congratulations 👶 ! A great project to start maintaining!

Thanks for the feedback. I must confess, the cookiecutter did the bulk of the heavy lifting, just filling in with your existing stuff.

Webpack is really quite nice for this kind of task. If you like, we can port the widget code to coffeescript with coffee-loader (my personal fav to-js syntax, though I wish it supported types, after trying typescript!). It's actually quite lovely to write widgets in (literate) coffee, as backbone plays so nice there :)

nicolaskruchten commented 8 years ago

All right, I've had a chance to dig into how this widget system works and I think I've got a good handle on it now.

It is indeed much nicer to be able to load all the JS and render elements directly in the Notebook DOM without an iframe/saving things to disk but the extra complexity on installation is a bit of a downer: for example just installing for development upgraded Jupyter for me from 4.1.0 to 4.2.1, I assume because of the ipywidgets>=5.1.5 dependency.

On thing which I'm disappointed by is that when I save my Notebook as HTML, the widget doesn't display... I had hoped that the widget system would make this "just work" similarly to how interactive Bokeh plots save to HTML without issue. Is there a trick to this, or a whole different approach?

bollwyvl commented 8 years ago

but the extra complexity on installation is a bit of a downer

Not to mention npm, et al. I hear you there. We're still working on making that simpler for all involved: developers, package maintainers, and of course notebook authors and their users.

4.1.0 to 4.2.1

Well, the big focus of 4.2 was making extensions more reasonable, providing more robust mechanisms for adding front-end and back-end components. Also, a lot of work went into the widgets during that time, as they are simultaneously supporting both the 4.x Notebook and new JupyterLab work. I don't know what to say there: it's non-trivial to support both, and in the case of conda/virtualenv, requires a tremendous amount of hackery to support 4.1.

save my Notebook as HTML, the widget doesn't display

That is still a work in progress. There is a mechanism for providing something which falls back to a CDN, but I don't think the big-ol-slug-of-embedded-js-and-css-and-fonts works yet, or is in the cards, but then I haven't done a lot of work around it. The new widgets do have some embedding functionality, but I haven't seen it work directly in the notebook JSON.

Bokeh plots save to HTML without issue. Is there a trick to this, or a whole different approach

Right, the bokeh approach is backed (by default) by CDN, but also has the ability actually embed its 1mb of code as HTML in the notebook. I don't know how best to reproduce this with the widgets.

nicolaskruchten commented 8 years ago

Hmm, ok. Here are some high-level thoughts...

Functional requirements

The primary use-case for this library is basically to as simply as possible dump some data from Pandas into an interactive control to quickly explore.
The reverse-path use-case (i.e. after UI interaction, get the reformated data back out) is less important to me, and I think it could be accomodated via a special Renderer (i.e. instead of a table, you get a textarea with a copy-pasteable pandas call to regenerate the same table as a dataframe?).
The sharing use-case (i.e. defining the pivot table parameters in Python and exporting to HTML) I value more than the reverse-path use-case.

Non-function requirements

ease of installation
writing to disk and/or relying on a CDN is brittle

Design tradeoffs

	current	widget	Bokeh embedding
primary use-case	yes	yes	yes
sharing use-case	no	no	yes
reverse-path use-case	no	yes	no
ease of installation	light	heavy	light
disk/cdn	bad	ok	ok

Based on this, I think I'm leaning more towards exploring a Bokeh-style embedding approach than a widget...

bollwyvl commented 8 years ago

I think sharing is within grasp, I just haven't done it yet. If nothing else, the webpack approach gives you the pattern for bringing all the dependencies along in one fell swoop.

As to installation: yeah, don't know what to do about that.

nicolaskruchten commented 8 years ago

Gotcha. I'll keep exploring and report back on the feasibility of the Bokeh approach. If I wanted to help make the sharing use-case possible, where could I contribute?

bollwyvl commented 8 years ago

What I mean by "haven't done it" is "haven't implemented widget embedding on any project," not just on this one.

Here's what the end result would look like:

http://jupyter.org/widgets.html

Crack open the javascript console/network panel and look at what loads up.

Basically, at runtime you reuse the widget base code, your widgets, and a minimal embedded state of the widgets. To do it, you do indeed have to use a CDN, though it first tries to grab the files locally. I don't know the story for doing it fully inline, a la bokeh... I've gone down this path for nbpresent (which doesn't use widgets, as its kernel-independent), and it is a GIANT PAIN, but necessary if true archival content is the goal (barring backwards-incompatible browser changes, I guess).

nicolaskruchten commented 8 years ago

OK, I understand what you're saying. From the linked page's section on the cookiecutter, though:

Following these practices will help make your custom widgets work in static web pages (like the examples of this page) and be compatible with future versions of Jupyter.

So maybe I just need to fiddle with the cookiecutter and the sharing use-case will work as well.

psychemedia commented 8 years ago

Just spotted this jupyter incubating project - a dataframe explorer widget that includes grouping, simple summary operators, and a few charts: http://blog.ibmjstart.net/2016/07/28/jupyter-declarativewidgets-make-data-exploration-easy/

fizcris commented 7 years ago

Congratulations on the tool. It is one of the most useful projects that I have seen after Pandas for Python.

I am a data scientist, not a programmer, so please don't shout at me if what I am saying is incoherent.

The tool is great for small amounts of data, but if we could couple it with SQL queries it would become much more interesting. (eg. Doing the filtering and selection of the data with the tool as it is and then plotting "x" scatter plots with "20"k points each, read temporarily from the DB/DataFrame.)

As a starting point just including another "Plot type" where you could run your own function would be the ideal. So it would take the filters applied as inputs and could return the plot in .html to fill in the plotting area.

nicolaskruchten commented 7 years ago

Glad you like this tool, @fizcris!

Your suggestions are very good and make a lot of sense, but unfortunately are not going to be possible to implement with the current architecture of this addon. This addon is basically just a tiny bit of glue to enable a notebook to pass data one time to the Javascript library PivotTable.js: there is no back and forth conversation between the pivot table controls and the data storage layer :)

psychemedia commented 4 years ago

There is an example of an ipywidget-ised version of pivottavle.js here: https://github.com/PierreMarion23/ipypivot

Another way of implementing pivottable.js as an ipywidget might be to wrap it using jp_proxy_widget (example of wrapping wavesurfer.js library).

stefaneidelloth commented 2 years ago

There is an example of an ipywidget-ised version of pivottavle.js here: https://github.com/PierreMarion23/ipypivot

Does not seem to work in JupyterLab 3.2.5: https://github.com/PierreMarion23/ipypivot/issues/12

psychemedia commented 2 years ago

@stefaneidelloth I wonder if the original js package could easily be wrapped using https://github.com/AaronWatters/jp_proxy_widget ?

nicolaskruchten / jupyter_pivottablejs