Interactive scribbling in notebook for image segmentation

[rsignell-usgs]

I spoke to some of you @EarthSim folks at SciPy2018, inquiring if pyviz/EarthSim tools could support the kind of "scribbling" interaction that we use to annotate images for deep neural network image segmentation of drone-derived beach photomosaics.

The work flow is that the user is presented with part of the image, and cycles through the different classes (e.g. sand, water, vegetation, etc) scribbling on each one to indicate the regions they see that match the class. A 2 minute video is here: https://www.youtube.com/watch?v=0sXYKmbLz_s, and an image is shown below where the user has already indicated sand regions and has moved on to water:

Would it be possible to use the existing pyviz/EarthSim tools to do this in a notebook cell, returning the scribbled line coordinates to python?

[jbednar]

Will the techniques from http://earthsim.pyviz.org/topics/GrabCut.html not work here?

[rsignell-usgs]

@jbednar , GrabCut is solving a similar problem, but with quite a different technique. This classification tool was developed @dbuscombe-usgs using pre-trained deep convolutional neural networks: http://www.mdpi.com/2076-3263/8/7/244

Here I'm really just asking if it would be possible to use pyviz to doodle on an image. I understand that I can already draw a segmented line by double clicking and then clicking to add each point, as in https://github.com/pyviz/EarthSim/blob/master/examples/user_guide/Annotators.ipynb

[philippjfr]

A freehand drawing tool is definitely on our to-do list but I haven't yet started work on adding it to bokeh.

[rsignell-usgs]

@philippjfr, "freehand drawing tool" is so much better than "scribbling" or "doodling"!
Should I raise the issue on bokeh issues, or not bother because it's already on the roadmap?

[philippjfr]

Should I raise the issue on bokeh issues?

Yes, that would be appreciated, I'll probably start on that very soon. It's already called out as a feature we want and should be a lot less complex than some of the other tools.

[rsignell-usgs]

Awesome! Great news. Closing in favor of https://github.com/bokeh/bokeh/issues/8133

[philippjfr]

Decided to have a go, here's a quick demo, PR incoming later today I hope:

[rsignell-usgs]

@philippjfr WHOA!!!! That's awesome!

[kcpevey]

The speed of @philippjfr puts the rest of us to shame! :)

[ceball]

Philipp Rudiger: PR by name, PR by nature :)

[felix-tracxpoint]

So how does one run it? :)

[philippjfr]

@felix-tracxpoint See the Drawing Tool user guide on the website. You'll have to follow install the dev version of the relevant packages including bokeh, holoviews and geoviews though.

[dbuscombe-usgs]

Thanks for this great addition. I finally got around to installing, following instructions posted here.

I would like to be able to freehand draw on an image without any geospatial information (i.e. just returning the 'image coordinates' of any annotations I make). Taking the provided image as an example, in examples/data, I can display the image like this:

import numpy as np
from imageio import imread
import holoviews as hv
from holoviews.streams import FreehandDraw
hv.extension('bokeh')

imfile = '../data/NewRiver_worldImageryRGB_20m.tif'
im = imread(imfile)

bounds=(0,0,1,1)   # Coordinate system: (left, bottom, top, right)
img = hv.Image(im, bounds=bounds)
img

but then I get stuck because I can't figure out how to add the FreehandDraw tool. I naively thought I could add a path like this

%%opts Path (line_width=1 color='white') [width=1 height=1] 
path = hv.Path([[(0, 0), (1, 1)]])
freehand_stream = FreehandDraw(source=path, num_objects=99)
bounds=(0,0,1,1)   # Coordinate system: (left, bottom, top, right)
img = hv.Image(im, bounds=bounds)
img * path 

but that didn't work. Any advice?

[philippjfr]

That works for me as long as you remove [width=1 height=1]. This sets the width and height of the plot to 1 pixel.

[dbuscombe-usgs]

So simple! Thanks @philippjfr !

Another thing, is there a way for the image to either 1) display much larger or 2) expand to another browser tab?

[dbuscombe-usgs]

Sorry, disregard. I guess that's what [width=1000, height=1000] does!