Towards replacing ImageView: outstanding issues

[timholy]

In this gist I have some code for creating 4d images and visualizing them with GLVisualize. (You need master on NRRD for this to work properly, plus 12G of free hard drive space for "big.raw".) You have to load a variable called img from either "small.nrrd" or "big.nhdr" first.

It's worth comparing these two files against ImageView. Here's a list of issues I see (I won't be surprised if some already have solutions):

Rendering issues:

• [x] When visualizing small.nrrd one sees that the pixel values have some kind of smoothing/blurring applied. I'm guessing there's some way to turn this off? And to choose a grayscale colormap? (Prob. should be the default for Gray{T} images?)
• [x] When visualizing big.nhdr I think I'm seeing some Mach banding

Usability issues:

• [x] When visualizing big.nhdr one can zoom in on the center region, but what's desperately needed is the ability to zoom in on an arbitrary region. ImageView/GtkUtilities has a nice feature where if you scroll a wheelmouse it zooms in/out around the position of the pointer, so all you have to do is point at the feature you're interested in and zoom in.
• [ ] For scientific applications, here's a common scenario: "from this graph I can see that something really weird happened in z-slice 23 at time-slice 3817. Let's go check it out." ImageView allows the user to type in specific times and z-positions into a text field. If this is hard to implement within the context of GLFW, perhaps we should consider a Gtk-based "UI controller" window? That said, I like the look of your UI controls better :smile:.
• [ ] For grayscale images, a purely grayscale contrast adjuster would be nice
• [ ] The exquisite control over the colormap is lovely, but sometimes a simple min=black/max=white is easier
• [ ] Maybe an interactive way to change the frame rate? Is that just a matter of creating another signal somehow?

A benefit is that performance on "big.nhdr" is oh-so-much better than ImageView.

[SimonDanisch]

Great feedback, thanks a lot!

1.1) Easy, but only exposed via the low level api right now. Also, you're using Intensity instead of Gray, which will displayed with a colormap. You can just directly display a gray image instead. 1.2) Intensity will per default have "levels", think of it as contourf. You can set linewidth to 0, or display it as a gray image.

Usability 2.1) yeah, i diverse set of camera tools would be nice... I have a couple of these implemented and flying around... I will try to put them together in your example! 2.2) I had this implemented numerous times but than updated my text rendering breaking text editing... I can see when we can get it back! Otherwise, I think @tknopp was interested in a GTK based UI as well. 2.3) That should be implemented manually. Hopefully GPUArrays will be mature enough soon, to have it operate at the same speed, as if it would be part of the shader (should even be faster, since it only needs to be done once per slider update, not once per frame) 2.4) I think this is 1.1/1.2 ? 2.5) That's correct ;) Just need to put a fps keyword somewhere and propagate it to the right signal...

[timholy]

Sounds promising. To clarify my point 2.4, what I mean is something like https://github.com/timholy/ImageView.jl/blob/master/readme_images/contrast.jpg. Seeing the histogram, and then positioning the "black level" and "white level", is a simple way to control contrast for many grayscale images (ones that may not be naturally scaled very well on the 0-to-1 scale).

[SimonDanisch]

Ah okay ;) So we'd need to implement a widget for that!

[timholy]

It occurs to me that it's not far from your current widget if there are only two control points and they are pegged to 0/1. The histogram would be nice, but perhaps not essential.

[tknopp]

Yes I would be very interested in getting GLVisualize working within a Gtk application.

[SimonDanisch]

Here we go: https://vimeo.com/206558295 I ticked off what seems to be solved by this ;) I replaced the big nrhd with some "more exciting" data, but failed at that ;) Some better widget for contrast control will follow! The code is still pretty messy, I'll follow up with a cleaner version (if you can't wait to test it, I can of course already give you access to this version)!

[timholy]

OMG. Amazing.

[timholy]

BTW, I'm starting on a GtkReactive.jl, just to make it easier to couple Gtk and GLVisualize. Just think, someday you'll be able to write a text document using Gtk's edit panes and then have GLVisualize running dynamic word clouds on it :smile:.

[tknopp]

@timholy would be amazing if we could embed the GLVisualize pane into an Gtk application. Actually most for this is in place since Gtk nowadays has an OpenGL widget which is already wrapped in Gtk.jl

[timholy]

Ideally we'd have GLVisualize render into a Gtk window. As you say, possible, but it may take some work on GLWindow to get this working. Before that happens, my goal is to get GLVisualize rendering into a GLFW window, but be controlled from a GUI written with Gtk (in a separate window). Not quite as elegant, but enough to get some real work done :smile:.

[tknopp]

yea, that sounds like a reasonable approach. Don't you get issues due to main loop clashes? For instance I cannot mix Gtk and PyPlot.

[SimonDanisch]

Oh really? On what level would they clash? I guess PyPlot clashes, since it also uses Gtk?

[tknopp]

No I think by default PyPlot does not use Gtk. Its some time that I tried this last but it simply gave me a segfault

[tknopp]

(Although this also might was due to the corrupted stack thingy we have in Gtk that I never fully understood why it cannot be properly fixed...)

[timholy]

I've been thinking about API. What do you think about

slct = sliceindex(A, 3)
A2 = slice(A, :, :, slct)

for positional dims and

slct = sliceindex(A, Axis{:time})
A2 = slice(A, slct)

for axisarrays. sliceindx creates a Signal for the index and GUI ctrls; slice is basically view except it allows GUI ctrls as indices. Even on the REPL you could push! to the signal and have values from getindex update. Makes it easy to do custom stuff with ctrls, like coupling two arrays together.

[tknopp]

@timholy is this Signaling concept you describe here orthogonal to what we have in Gtk were I would simply use signal_connect and then would do the slicing in a callback?

[timholy]

See Reactive.jl. It took me a while to wrap my head around this style of programming (and I'm still learning), but it does seem to be a nice way to organize callbacks. Moreover, GLVisualize uses Reactive signals for everything, so to get the two talking together the first order of business is to link Gtk signals to Reactive signals. Already in progress on the dev branch of GtkReactive (which will move elsewhere once it's ready for prime time).

[SimonDanisch]

What problem are you trying to solve?

[SimonDanisch]

I'd love to have the same API for any Gtk/GLVisualize gui combinations... So that you could just do something like:

import GLVisualize: slider
# import Gtk: slider # interchangeable 
...
x = slider(1:10)
...

[tknopp]

This is unfortunately not practical. In Gtk you do not really want to do the layout of widgets in code (I know Tim will disagree with me). What you do is having the UI defined in xml and load this in your code. This is how most toolkits work (Gtk, Qt, WPF, ...)

I have written a very complex UI in Gtk.jl and can say that it is really important to not mix the Model/Controller and the View (MVC concept)

[timholy]

I'd love to have the same API for any Gtk/GLVisualize gui combinations...

Yes, I'm aiming to match Interact.jl names.

What problem are you trying to solve?

Ensuring that interesting stuff isn't buried deep inside some graphics-specific toolkit in a way that users can't get their hands on things. For example, suppose I have two 3d arrays, one of a brain MRI and the other a segmented version of the same image, using colors to encode the different regions. Now I want to make a GUI that shows both images in two panes, linking the two to a single control for slice-selection. That will look something like this:

zslice = Signal(1)
imga = view(mri, :, :, zslice)       # imga is now an AbstractMatrix, but with a Reactive Signal for the chosen slice
imgb = view(segmented, :, :, zslice) # using the same Signal, so they are linked
cnvsa = imshow(imga) 
cnvsb = imshow(imgb)
preserve(map(x->draw(cnvsa), zslice))
preserve(map(x->draw(cnvsb), zslice))
nav = playergui(zslice)  # creates a "player" widget

Even for 2d images, I think the following will be super-cool (this may not be consistent with the "make everything a camera manipulation" viewpoint of GLVisualize, though):

zoomy, zoomx = Signal(indices(img,1)), Signal(indices(img,2))
imgz = view(img, zoomy, zoomx)
imshow(imgz, zoom = (zoomy, zoomx))
# Now, the user zooms in on a particular feature using a rubber-band selection
mean(imgz)
# --> computes the mean intensity/color over just the zoomed-in region!

How awesome is that?

[timholy]

This is unfortunately not practical. In Gtk you do not really want to do the layout of widgets in code (I know Tim will disagree with me).

I don't disagree, actually. I think it's completely orthogonal to layout, and in fact I plan on omitting anything related to layout from GtkReactive. It will merely be an issue of allowing the user to specify a pre-existing GtkWidget instead of creating one from scratch.

Really, you can almost think about Signals as Ref variables:

slidervalue = Ref(1.0)
signal_connect(myslider, :value_changed) do widget
    slidervalue[] = G_.value(widget)
end

except instead by push!ing to a Signal you can automatically trigger other actions, like redrawing a canvas.

[tknopp]

yes this sounds like a clever plan. I was referring to Simons example

x = slicer(1:10)

In my Gtk code this never happens. I get the slider from the GtkBuilder object. So the slider is already constructed.

[SimonDanisch]

I see, makes sense ;) I guess my scenario is more geared towards casual creation of a throw away GUI ;)

[tknopp]

kind of. I have written all my first prototypes in Code and then always came to that situations where I wanted to reorganize the layout and this is painful. I should add that the UI can be put together using graphical tools (i.e. Glade in case of Gtk.jl). This made me much more productive.

The thing that Tim wants to push reminds me a lot on "Data Bindings" concept from WPF/C#. Have used that when working on a large application and it really is a nice concept.

[timholy]

I think we want to support both models. Fortunately, that's really easy to do.

[SimonDanisch]

Obviously I hadn't much time to improve this, but I at least moved it into a module and made sure that corner cases work: https://github.com/SimonDanisch/GLPlot.jl/blob/master/example/imageview.jl

If you need more comments to hack on it, let me know!

[timholy]

Sounds like you have a big agenda. No worries, I'm excited about what you're working on!