an offer of an small intermediate tutorial ipywidget notebook & nbextension example using d3

[paul-shannon]

In preparation for creating a cytoscape.js ipywidget as an nbextension, I figured I would create a simple stripped down interactive widget using another better-known and popular js library - d3 - as a good learning exercise.

I began this process by studying https://ipywidgets.readthedocs.io/en/latest/ Then I looked at the three exemplary (but complicated) nbextension widgets: bqplot, pythreejs, ipyleaflet. Impressive, but too complicated for my currently weak understanding.

After many hours of hacking around, experimenting, and web searching, I finally have a none-too-pretty demo notebook which draws an interactive d3 circle on an svg canvas, written asDOMWidget and DOMWidgetView subclasses. The circle changes color on mouseover. Nothing grand here.

If I developed this example, kept it small, made it somewhat more interesting, used only best practices for ipywidgets and nbextensions, and created a well-documented notebook that could easily be moved into a proper cookiecutter nbextensions template, and eventually into jupyter lab, would you

• find this useful?
• be willing to critique my work?
• answer questions?
• consider a PR, a mid-level tutorial example, falling somewhere between datepicker and the three more advanced examples (pythreejs, bqplot, ipyleaflet)?

Thank you,

• Paul Shannon Institute for Systems Biology Seattle

[dmadeka]

By the way @paul-shannon , just as an fyi simply doing

from bqplot import pyplot as plt
plt.figure()
plt.scatter([0.5], [0.5])
plt.show()

should do exactly the same thing as the example of appending a d3 circle. You can also add the exact same mouse over behavior with little effort. Not sure why you say it's complicated... just wanted to know

[paul-shannon]

Hi Dhruv,

It is not that the USE of bqplot is complicated. As you illustrate, it is nice and simple.

But for the sake of learning, my experience is that it is pretty complicated, not transparent to a relative beginner programming in these idioms. I see 15,900 lines of javascript code, 4300 lines of python.

• Paul

On Oct 17, 2016, at 4:49 PM, Dhruv Madeka notifications@github.com wrote:

By the way @paul-shannon , just as an fyi simply doing

from bqplot import pyplot as plt plt.figure() plt.scatter([ 0.5], [0.5 ]) plt.show()

should do exactly the same thing as the example of appending a d3 circle. You can also add the exact same mouse over behavior with little effort. Not sure why you say it's complicated... just wanted to know

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jasongrout]

If I developed this example, kept it small, made it somewhat more interesting, used only best practices for ipywidgets and nbextensions, and created a well-documented notebook that could easily be moved into a proper cookiecutter nbextensions template, and eventually into jupyter lab, would you

find this useful?

Yes!

be willing to critique my work?

Definitely!

answer questions?

Certainly!

consider a PR, a mid-level tutorial example, falling somewhere between datepicker and the three more advanced examples (pythreejs, bqplot, ipyleaflet)?

That sounds like a fantastic idea!

[paul-shannon]

Great to get this response. Thank you.

My first version will be, I think, embarrassingly bad. Shall I start, therefore, with a gist, not a PR?

• Paul

On Oct 17, 2016, at 5:28 PM, Jason Grout notifications@github.com wrote:

If I developed this example, kept it small, made it somewhat more interesting, used only best practices for ipywidgets and nbextensions, and created a well-documented notebook that could easily be moved into a proper cookiecutter nbextensions template, and eventually into jupyter lab, would you

find this useful?

Yes!

be willing to critique my work?

Definitely!

answer questions?

Certainly!

consider a PR, a mid-level tutorial example, falling somewhere between datepicker and the three more advanced examples (pythreejs, bqplot, ipyleaflet)?

That sounds like a fantastic idea!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[paul-shannon]

I will be glad to submit in any mode your prefer, for your review, critique and suggestions. For starters, however, and hoping this is not inconvenient, my first solution (flawed...) is

https://github.com/paul-shannon/notebooks/blob/master/study/CircleView.ipynb

I will be grateful for any and all comments.

[jasongrout]

I think the d3 requirement should go at the top, with the jupyter-js-widgets requirement:

define('circle', ["jupyter-js-widgets", "d3"], function(widgets, d3) {

[jasongrout]

Also, you probably don't need to create the svg element every time the ratio is changed. I think creating it once in the render, and then using d3 to modify the svg element would be cleaner.

[jasongrout]

Overall, good work!

[paul-shannon]

I spent hours yesterday trying to make that approach work - that is, using the standard requirejs define statement with both jupyter-js-widgets and d3. Why I failed, I may never know...

It works perfectly now, just as you predict.

Thank you!

• Paul

On Oct 19, 2016, at 10:23 AM, Jason Grout notifications@github.com wrote:

I think the d3 requirement should go at the top, with the jupyter-js-widgets requirement:

define('circle', ["jupyter-js-widgets", "d3"], function(widgets, d3) {

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[paul-shannon]

Maintaining the svg, creating it only once, is clearly the right thing to do. There are now three separate methods:

createDiv createCanvas drawCircle

It took a while to figure out: the svg canvas cannot be created until the underlying div appears in the DOM, which takes a little time. I now use a setTimeout to do that; some sort of waitForElement would be more robust. Any suggestions?

Two other topics have me puzzled. I describe them, in context in the updated notebook and here as well.

https://github.com/paul-shannon/notebooks/blob/master/study/CircleView.ipynb

1) I use window.svg rather this.svg to preserve the svg reference between calls to drawCircle. I can preserve an integer field (e.g., this.testCounter) but not this.svg, nor this.svgCreated.

Is the “this” reference (a CircleView object, which extends widgets.DOMWidgetView) managed by the base class in a way that limits the creation of new slots? Clearly, assignments to the browser global “window” are a bad idea.

2) Where can I read up on how to extend my python widgets.DOMwidget class to support a constructor and methods like these?

cw = CircleWidget(height=300, width=500) cw.drawCircle(radius=30, x=100, y=100, borderColor="red", fillColor="white”)

Many thanks!

• Paul

On Oct 19, 2016, at 11:29 AM, Jason Grout notifications@github.com wrote:

Also, you probably don't need to create the svg element every time the ratio is changed. I think creating it once in the render, and then using d3 to modify the svg element would be cleaner.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jasongrout]

1) I use window.svg rather this.svg to preserve the svg reference between calls to drawCircle. I can preserve an integer field (e.g., this.testCounter) but not this.svg, nor this.svgCreated.

Is the “this” reference (a CircleView object, which extends widgets.DOMWidgetView) managed by the base class in a way that limits the creation of new slots? Clearly, assignments to the browser global “window” are a bad idea.

Using the global "window.svg" means that there can be only one of these on the page (otherwise they overwrite each other). You're right that this is a bad idea :).

The d3.select can take a DOM node directly - no need to look it up on the page. I would probably do something like:

d3.select(this.el).append('div').append('svg')

instead of creating the div and svg separately. Save a reference to the svg in this.svg, and probably save a reference to the circle as well. Then all you have to do is adjust the circle's r attribute in _radius_changed, rather than having to create a new circle element.

2) Where can I read up on how to extend my python widgets.DOMwidget class to support a constructor and methods like these?

cw = CircleWidget(height=300, width=500) cw.drawCircle(radius=30, x=100, y=100, borderColor="red", fillColor="white”)

You can automatically pass in any arguments that are attributes of the python class. Right now you have ratio. Do the same thing you've done for ratio for these other attributes.

[paul-shannon]

Where do I look for docs on the javascript widgets.DOMWidgetView listenTo function? I am unable to detect, when using a List(CInt()) traitlet, the python event in which that list is appended to.

This now partly works, but incompletely, still puzzling. Defining my python class like this:

class CircleWidget(widgets.DOMWidget):
    _view_name = Unicode('CircleView').tag(sync=True)
    _view_module = Unicode('circle').tag(sync=True)
    circles = List(trait=Int()).tag(sync=True)
    def drawCircle(self, newRadius):
       #  self.circles.append(newRadius)  # no change event in js
       self.circles = [newRadius, newRadius-1]  # change event seen, data available in js

javascript excerpt:

   render: function() { 
        this.$el.append(this.createDiv());
        this.listenTo(this.model, 'change:circles', this._circles_changed, this);
        setTimeout(this.createCanvas, 500);
        },

    _circles_changed: function() {
       console.log("_circles_changed")
       console.log(this.model.get('circles'));
       }

Appending to self.circles in python does not appear to register as a true change to the traitlet system, whereas an explicit assignment of a fresh list does.

Complete notebook here:

https://github.com/paul-shannon/notebooks/blob/master/study/CircleView.ipynb

I'll be grateful for help, and pointers on what to read up on .

• Paul

[jasongrout]

Where do I look for docs on the javascript widgets.DOMWidgetView listenTo function? I am unable to detect, when using a List(CInt()) traitlet, the python event in which that list is appended to.

The traitlets List does not support events for modifying the list, such as append. We usually use the traitlets Tuple type to make that clear. It only supports events for reassignment, and only if a new list is assigned.

[paul-shannon]

Good tip, thanks. I can now construct circles python, prepend them to a fresh tuple for the traitlet, see the resulting list of objects in javascript.

In order to grok the full range of sync operations ipywidgets provide, I support circle creation in javascript also, from interactive click on the d3 canvas, adding an object for that new circle to the locally maintained array. But the syncing of this array back to python does not yet work - maybe because the python traitlet is that immutable tuple?

circleView.model.set("circles", JSON.stringify(circleView.circles));  
this.touch()

What do you recommend for creating (and syncing) a list of objects (circles, in my case) both ways, back and forth between python and javascript? Can I register a python function to receive the incoming list, forge a new tuple?

CircleView notebook

[paul-shannon]

Offering my own followup here, and a tangential question:

I see now that I need a pretty good understanding of backbone.js before I proceed. I will be back to this thread in a week or so after I have done that.

Is anyone working on an R implementation of juyter js widgets?

Thanks for helping me get this far.

• Paul

[jasongrout]

Is anyone working on an R implementation of juyter js widgets?

Not that I know of. That would be awesome if someone were - we'd love to support R notebooks with widgets.

[paul-shannon]

The traitlets List does not support events for modifying the list, such as append. We usually use the traitlets Tuple type to make that clear. It only supports events for reassignment, and only if a new list is assigned.

In order to exchange more complex data types between the browser and the python kernel, and with Lists and Tuples quite constrained, I send json objects in mutable Unicode strings back and forth, parsing them into language-appropriate variables on each end.

This is the only strategy I have figured out for when manipulating state (of structured data) equally in each environment is a goal, - that is, in both python and javascript, and keeping them in sync.

An example: in a cytoscape.js canvas, one might delete a subgraph, and want this reflected back to the python kernel igraph data structure.

Is there a better approach?

[paul-shannon]

we'd love to support R notebooks with widgets.

This is also a strong interest of mine. I spent a few years with Bioconductor, so I might be a plausible contributor.

Have you any suggestions of how I can explore this possibility?

[paul-shannon]

Two repos, both present a simple jupyter widget: a d3 canvas on which circles are drawn in response to python function calls, or by directly clicking on the d3 canvas.

The first demo (possibly useful as a tutorial also) is simple, with all code self-contained in one notebook:

https://github.com/paul-shannon/jupyter-widget-demo-all-in-notebook

The second demo - to my mind, pretty complex - should be useful if you want to learn how to make installable notebook widget extensions and have little prior understanding of the mechanisms involved:

https://github.com/paul-shannon/jupyter-widget-demo-nbextension

Suggestions, advice, criticism: all are welcome.

• Paul

[jasongrout]

Thanks Paul, and sorry for the long-delayed feedback!

The first tutorial looks great.

The second also looks good (though I haven't tried it out yet). It's certainly more complicated, like you noted. I also noticed that the github repo has many files that are not typically included in a git project repo (for example, the node_modules directory, the RCS directories, etc.). Did you mean to include all of those?

[jasongrout]

we'd love to support R notebooks with widgets.

This is also a strong interest of mine. I spent a few years with Bioconductor, so I might be a plausible contributor.

Have you any suggestions of how I can explore this possibility?

The place to start would be implementing the comm communication mechanism in the R kernel if it's not implemented already (https://jupyter-client.readthedocs.io/en/stable/messaging.html#custom-messages), followed by implementing the equivalent to the python ipywidgets code sending the appropriate widget messages (see https://github.com/jupyter-widgets/ipywidgets/blob/master/packages/schema/messages.md, protocol version 2 for ipywidgets 7)

[jasongrout]

In order to exchange more complex data types between the browser and the python kernel, and with Lists and Tuples quite constrained, I send json objects in mutable Unicode strings back and forth, parsing them into language-appropriate variables on each end.

This is the only strategy I have figured out for when manipulating state (of structured data) equally in each environment is a goal, - that is, in both python and javascript, and keeping them in sync.

An example: in a cytoscape.js canvas, one might delete a subgraph, and want this reflected back to the python kernel igraph data structure.

Is there a better approach?

You could use an Instance traitlet to encapsulate the kernel igraph data structure, and then have custom serializers that serialize and deserialize its representation to JSON for syncing. That's how we work with numpy arrays in bqplot or ipyvolume, for example.

[paul-shannon]

Hi Jason,

Good to hear back.

Those extraneous files are a result of me being somewhat clueless about .gitignore at the time.

If these tutorials fit - perhaps after appropriate revision - with the new ipython widget, I would be glad to update them.

Related: I still hope to see that R notebooks/labbooks/cells can make similar use of the ipython widget infrastructure. I would be glad to contribute if I could be helpful.

I will be at jupytercon in NYC. Perhaps our paths will cross there, and you can steer me in useful directions.

• Paul

On Aug 3, 2017, at 5:44 AM, Jason Grout notifications@github.com wrote:

Thanks Paul, and sorry for the long-delayed feedback!

The first tutorial looks great.

The second also looks good (though I haven't tried it out yet). It's certainly more complicated, like you noted. I also noticed that the github repo has many files that are not typically included in a git project repo (for example, the node_modules directory, the RCS directories, etc.). Did you mean to include all of those?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jasongrout]

I will be at jupytercon in NYC. Perhaps our paths will cross there, and you can steer me in useful directions.

Great! Are you going to be there for the Sprint on Saturday? That might be a good time to work on this.

[paul-shannon]

Alas, I fly home early Saturday. Perhaps you can suggest another time where maybe 10 minutes discussion could set me off in a good direction?

• Paul

On Aug 3, 2017, at 10:35 AM, Jason Grout notifications@github.com wrote:

I will be at jupytercon in NYC. Perhaps our paths will cross there, and you can steer me in useful directions.

Great! Are you going to be there for the Sprint on Saturday? That might be a good time to work on this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jasongrout]

I'm sure we'll find a time for a 10-minute discussion.