[moved] Draw polygons on a map from a notebook

[alvinwan]

Prof. DeNero

I think we want something like https://github.com/pbugnion/gmaps to draw a Google map with polygons & dots on it using the Google Maps Javascript API. It's unclear whether this library has been ported to Python 3. They have a directory that gives me hope: https://github.com/pbugnion/gmaps/tree/master/examples/ipy3

[SamLau95]

The gmaps package was written in Python 2 so imports were broken all over the place when installing using Python 3. I made a fork that seems like it works okay at https://github.com/SamLau95/gmaps

(As an aside, I put it on my personal GH since I don't have the ability to create repos on this org. @papajohn was that intentional?)

[SamLau95]

From what I can tell, the gmaps package doesn't have too much power. In fact, it can only make heatmaps, for the most part. But maybe these are good enough for what we're looking to do.

[alvinwan]

Judging by project 1's title, it seems like we need the polygon to wrap around an area.

gmaps is pretty simple, but it seems like a good guide so far.

[alvinwan]

Didn't know about the 2to3 tool either. That's pretty neat, what you did there.

[papajohn]

We might have to write our own Python wrapper for the Google Maps Javascript API that works in the IPython environment. I think we need points and polygons more than heat maps. Hopefully gmaps can get us started quickly.

On Fri, Jul 17, 2015 at 8:06 PM, Alvin Wan notifications@github.com wrote:

Didn't know about the 2to3 tool either. That's pretty neat, what you did there.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-122477228.

[deculler]

That's pretty much where I came to as well. Ought to be able to use google maps as a background image for basemap in matplotlib.

David E. Culler Friesen Professor of Computer Science Electrical Engineering and Computer Sciences University of California, Berkeley

On Fri, Jul 17, 2015 at 10:36 PM, John DeNero notifications@github.com wrote:

We might have to write our own Python wrapper for the Google Maps Javascript API that works in the IPython environment. I think we need points and polygons more than heat maps. Hopefully gmaps can get us started quickly.

On Fri, Jul 17, 2015 at 8:06 PM, Alvin Wan notifications@github.com wrote:

Didn't know about the 2to3 tool either. That's pretty neat, what you did there.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-122477228.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-122485428.

[papajohn]

I have an hour to kill. Any suggestions about how I can make progress on this issue? I'll start by reading the Google Maps Javascript API.

[SamLau95]

I'm going to be working on this later tonight. I think that's a good place to start; any documentation / recommendations on that would save us time searching.

[papajohn]

I'm sending everyone an API key over email, since this repo is public. Feel free to use it as much as you need.

The Maps Embed API is not sufficient for our needs. We have to use the Javascript API.

On Mon, Jul 20, 2015 at 5:15 PM, Sam Lau notifications@github.com wrote:

I'm going to be working on this later tonight. I think that's a good place to start; any documentation / recommendations on that would save us time searching.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123103188.

[alvinwan]

@papajohn @SamLau95

Here's how I broke down the application.

IPython integration:

• I found IPython line-and-cell magics helpful. I was originally thinking that all commands would go into this magic.

Generating a map, using Javascript:

• magic injects a separate script
• The JS API is limited to points, I believe. We could settle with dotted lines?

Generating a map, using Python:

• magic is directly invokes a python method
• matplotlib was interesting, although I didn't dig deep enough to see if those are helpful http://matplotlib.org/basemap/. Again, only points.

I'll check back soon, to see if you guys need another hand... once I'm done with ok-IPython integration.

[alvinwan]

@papajohn Sounds good.

[papajohn]

It looks like we can add arbitrary shapes to the map using the JS API: https://developers.google.com/maps/documentation/javascript/shapes

On Mon, Jul 20, 2015 at 5:27 PM, Alvin Wan notifications@github.com wrote:

@papajohn https://github.com/papajohn @SamLau95 https://github.com/SamLau95

Here's how I broke down the application.

IPython integration:

• I found IPython line-and-cell magics helpful. I was originally thinking that all commands would go into this magic.

Generating a map, using Javascript:

• magic injects a separate script
• The JS API is limited to points, I believe. We could settle with dotted lines?

Generating a map, using Python:

• magic is directly invokes a python method
• matplotlib was interesting, although I didn't dig deep enough to see if those are helpful http://matplotlib.org/basemap/. Again, only points.

I'll check back soon, to see if you guys need another hand... once I'm done with ok-IPython integration.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123107306.

[papajohn]

@alvinwan if you could get us started on this IPython magics business, that would be helpful. I think you have a PR, but there's no javascript checked in. Did you write some already, perhaps? If not, do you know how the interface to IPython works? Thanks!

[alvinwan]

Whoa, that's a nice feature you found.

I can work on the magic. I think I've hit a wall with IPython-ok integration anyhow. I just commented on the PR, but basically, I think the magic will output HTML+JS:

1. I can work the magic (hah) so that there's a generic way to pass flags/args to JS.
2. You could then receive those flags/args and build the map, assuming you have all the HTML+JS you need?

Naturally, part 2 sounds more time-consuming/difficult, so I could help with that as well, once we're happy with the magics.

[alvinwan]

*I didn't actually write any JS, so nothing to preserve. I can scrape something together if that'll give you some headway though.

[papajohn]

Yes, if you can configure us to start making JS API calls from Python somehow, then I think we'll make fast progress. That is magic.

On Mon, Jul 20, 2015 at 5:41 PM, Alvin Wan notifications@github.com wrote:

Whoa, that's a nice feature you found.

I can work on the magic. I think I've hit a wall with IPython-ok integration anyhow. I just commented on the PR, but basically, I think the magic will output HTML+JS:

1. I can work the magic (hah) so that there's a generic way to pass flags/args to JS.
2. You could then receive those flags/args and build the map, assuming you have all the HTML+JS you need?

Naturally, part 2 sounds more time-consuming/difficult, so I could help with that as well, once we're happy with the magics.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123110965.

[papajohn]

I guess I'd prefer to keep as much in Python as possible and just add whatever JS we need to make API calls (based on arguments defined in Python) and display the results. I think that's possible, but I haven't read enough of the IPython docs to understand how to do it yet.

On Mon, Jul 20, 2015 at 5:42 PM, Alvin Wan notifications@github.com wrote:

*I didn't actually write any JS, so nothing to preserve. I can scrape something together if that'll give you some headway though.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123111161.

[alvinwan]

haha x) And, I see. It definitely is... but I don't know if it's a "clean" way of doing it. The premise:

1. line magic spits out JS that calls the JS API accordingly.
2. JS invokes python via the IPython kernel.
3. voila - Python now has the results of a JS API call.

Although, I was thinking that Python simply passes arguments, and the JS handles the load. This way, we don't need the extra step (JS -> back to -> Python).

What should the final product do? (take a place as an arg and make a polygon around it?)

[alvinwan]

*This link explains the JS/Python connection: https://jakevdp.github.io/blog/2013/06/01/ipython-notebook-javascript-python-communication/

[papajohn]

Here's a spec of the one Python function that we want to implement. I think that all of its arguments & its behavior correspond to stuff in the Google Maps JS API.

def draw_map(center, zoom, points=[], regions=[]):
    """Draw a map with center & zoom containing all points and 
    regions that displays points as circles and regions as polygons.

    center -- lat-long pair at the center of the map
    zoom -- zoom level
    points -- a sequence of MapPoints
    regions -- a sequence of MapRegions
    """
    # Some magic to get javascript to display the map

class MapPoint:
    """A circle https://developers.google.com/maps/documentation/javascript/shapes#circles"""
    def __init__(self, center, radius, strokeColor, strokeOpacity, strokeWeight, fillColor, fillOpacity):
        ...

class MapRegion:
    """A polygon https://developers.google.com/maps/documentation/javascript/shapes#polygons"""
    def __init__(self, paths, strokeColor, strokeOpacity, strokeWeight, fillColor, fillOpacity):
        """paths -- a list of lat-long pairs or a list of list of lat-long pairs"""
        ...

[papajohn]

@alvinwan Should I be nervous that the blog post you linked starts with, "the functionality I'll outline here is planned to be obsolete in the 2.0 release of IPython."

[alvinwan]

I see I see. I think the Python abstraction can generate JS output, which is then sent back to the notebook.

haha I completely missed that, but that's how jupyter-notebook-student invokes ok as of this moment. Not all of it went away, I guess.

[papajohn]

One tricky thing to keep in mind. The JS API expects every lat-long pair to be an object. There's a lot of google.maps.LatLng(x, y) in the examples. So, the JS layer will have to perform this type coercion in addition to invoking the API in the right way and inserting the resulting map in the right place.

[papajohn]

I might be wrong, but I think since there are no interactive controls, Javascript never needs to invoke Python. Instead, executing the Python draw_map function should invoke some javascript that renders the result. Once the result is on the screen, it's self-contained (even though you can drag/rescale the map, Python never knows that you did).

[alvinwan]

Yeah, no you're right. I misinterpreted.

I just outlined a plan of attack in the PR (updated the original post) - could you take a look? Basically, the notebook invokes Python which generates Javascript. Development will (kind of) go in the reverse order though; get a simple Javascript sample working, make a Python wrapper, and finally, integrate with the notebook.

[SamLau95]

From what I can tell, @alvinwan is right. It actually seems relatively simple, no magic is needed on our end (IPython does the connection between Python and JS).

Take a look at the plainmap Python code from the gmaps module. On line 8, the _view_name corresponds to the Javascript PlainmapView which extends an IPython DOMWidgetView. This this a backbone.js view, which means that the render function gets called to render the map. The attributes from Python are made available to the JS in this.model (see lines 13/14 in the JS which read the attributes in lines 14/15 in Python).

Since John's figured out the attributes we want to pass from Python to Javascript, this problem is now reduced to figuring out how to work with the GMaps JS API to get what we want and connecting with the IPython widget on the python end, which IPython makes easy for us through its WidgetManager

@alvinwan: Have you already started working on this? I can jump in right now if not. @papajohn: Are we making these additions in the gmaps repo on my personal github? Or integrating them in this repo?

[alvinwan]

@SamLau95 I haven't started. Go for it. Working on ok and will transition back to this later today or tomorrow.

Your explanation makes a lot of sense; I never really understood what gmaps was doing. Do you know what the "traitlets" are? Do they just govern the widget's display?

On the same note, I think the goal is to ultimately include our implementation of gmaps in the datascience pip package, which would mean building it from scratch or salvaging the parts we want, for this repo.

[deculler]

The other avenue is you use the map as a background in basemap and use basemap to draw on it.

David E. Culler Friesen Professor of Computer Science Electrical Engineering and Computer Sciences University of California, Berkeley

On Mon, Jul 20, 2015 at 5:29 PM, John DeNero notifications@github.com wrote:

It looks like we can add arbitrary shapes to the map using the JS API: https://developers.google.com/maps/documentation/javascript/shapes

On Mon, Jul 20, 2015 at 5:27 PM, Alvin Wan notifications@github.com wrote:

@papajohn https://github.com/papajohn @SamLau95 https://github.com/SamLau95

Here's how I broke down the application.

IPython integration:

• I found IPython line-and-cell magics helpful. I was originally thinking that all commands would go into this magic.

Generating a map, using Javascript:

• magic injects a separate script
• The JS API is limited to points, I believe. We could settle with dotted lines?

Generating a map, using Python:

• magic is directly invokes a python method
• matplotlib was interesting, although I didn't dig deep enough to see if those are helpful http://matplotlib.org/basemap/. Again, only points.

I'll check back soon, to see if you guys need another hand... once I'm done with ok-IPython integration.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123107306.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123108186.

[alvinwan]

@deculler I finally understood what you meant. matplotlib ref.

We would probably have to manipulate the canvas directly though e.g., snapshot a png, layer strokes on it, flatten canvas

[SamLau95]

@alvinwan Traitlets are wrappers around attributes that let IPython know to share em between Python and JS (see third paragraph of https://github.com/ipython/ipython/wiki/IPEP-23:-Backbone.js-Widgets#description)

[papajohn]

I like the fact that Google maps let you scroll & zoom. They're also prettier and more familiar than any of the basemap examples.

I'd like this implementation to end up in the https://github.com/dsten/datascience repo. It's easier to deploy that way (using pip install datascience).

Thanks for working on this, all! It sounds like Sam has a good idea of what to build; reassigning to him for a while. Let us know if we can help.

On Mon, Jul 20, 2015 at 9:47 PM, Sam Lau notifications@github.com wrote:

@alvinwan https://github.com/alvinwan Traitlets are wrappers around attributes that let IPython know to share it between Python and JS (see third paragraph of https://github.com/ipython/ipython/wiki/IPEP-23:-Backbone.js-Widgets#description )

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123157574.

[alvinwan]

@SamLau95 Feel free to scrap or just ignore the old PR I started. I won't be hurt.

[papajohn]

By the way, I'm in meetings straight from 9 to 6 tomorrow, so won't be much help, but one of those meetings (11am) is with a geo data specialist. I'll ask her if we're missing anything important in our proposed approach to map visualization.

[SamLau95]

@alvinwan Looked it over. I'm basically doing the same approach without using magic functions so I'm taking your setup.

[alvinwan]

Sounds good. Let me know if you could use an extra hand.

[papajohn]

I think we're potentially on the wrong track. The Geo specialists recommended that we explore

https://github.com/python-visualization/folium

They are going to send us some examples. We'll see how well it fits into our current setup. We wouldn't have to wrap a JS library as a result, and perhaps most importantly, this folium package does a good job of drawing legends while Google Maps doesn't.

[alvinwan]

Wow, looking forward to the examples.

[deculler]

Yes, folium is much more like to fit with matplotlib. cartopy and iris are other important possibilities.

David E. Culler Friesen Professor of Computer Science Electrical Engineering and Computer Sciences University of California, Berkeley

On Tue, Jul 21, 2015 at 11:51 AM, John DeNero notifications@github.com wrote:

I think we're potentially on the wrong track. The Geo specialists recommended that we explore

https://github.com/python-visualization/folium

They are going to send us some examples. We'll see how well it fits into our current setup. We wouldn't have to wrap a JS library as a result, and perhaps most importantly, this folium package does a good job of drawing legends while Google Maps doesn't.

— Reply to this email directly or view it on GitHub https://github.com/dsten/datascience/issues/5#issuecomment-123439701.