trying to set up #uicontrol as default

[falkben]

I am trying to transition NeuroData's color and min/max ui controls to use the new #uicontrol which seems better and easier to make compatible with the different neuroglancer deployments around.

The problem I am having is that it seems as though the variables defined inside the #uicontrol don't aren't initialized in time for the shader, so the shader errors out with a bunch of "undeclared identifier" messages. Once you edit the shader, everything starts to work again.

Here's what I have:

I put the following shader as a default inside src/neuroglancer/sliceview/volume/image_renderlayer.ts:

#uicontrol vec3 color color(default="white")
#uicontrol float min slider(default=0, min=0, max=1, step=0.01)
#uicontrol float max slider(default=1, min=0, max=1, step=0.01)
#uicontrol float brightness slider(default=0, min=-1, max=1, step=0.1)
#uicontrol float contrast slider(default=0, min=-3, max=3, step=0.1)

float scale(float x, float min, float max) {
  return (x - min) / (max - min);
}

void main() {
  emitRGB(
    color * vec3(
      scale(toNormalized(getDataValue()), min, max) + brightness,
      scale(toNormalized(getDataValue()), min, max) + brightness,
      scale(toNormalized(getDataValue()), min, max) + brightness
    ) * exp(contrast)
  );
}

I also have a little bit of logic in src/neuroglancer/image_user_layer.ts order to make old ndviz links compatible with the new system (automatically pulling in min/max/color and putting them into the shader_controls).

The branch is located here: https://github.com/neurodata/neuroglancer/tree/ben/preserve-color-state-uicontrol

I've been struggling for a bit, and wondering if the #uicontrol are really only meant to be something to be interacted with after a channel is loaded or whether a customized default #uicontrol makes sense.

Thanks!

[jbms]

Yes, this is exactly the sort of thing I intended to be possible --- and was considering making something like this the default shader in neuroglancer upstream. I noticed some bugs in the handling of the shader UI controls by image_renderlayer.ts previously while working on the ndims branch and had fixed the issues there. I just cherry picked those changes over to the master branch, which resolves this particular issue.

On the topic of min/max values, however, I was actually thinking of going in the opposite direction and adding special support to the image layer for min/max values like neurodata already does: specifically integrating the min/max controls into a histogram of the data values in the currently visible chunks --- particularly for 16-bit and float data I imagine a histogram will be very useful for picking reasonable min/max values. However, I suppose the histogram+range selection could just be available as a shader control as well.

Even if the histogram-based selection is available as a shader control, it might still be sense though to also have a built-in min/max control that either affects toNormalized, or affects e.g. a new getNormalizedDataValue function.

Note: for your shader, you can just multiply the color by a float, no need to repeat the scale expression 3 times.

[falkben]

On the topic of min/max values, however, I was actually thinking of going in the opposite direction and adding special support to the image layer for min/max values like neurodata already does: specifically integrating the min/max controls into a histogram of the data values in the currently visible chunks --- particularly for 16-bit and float data I imagine a histogram will be very useful for picking reasonable min/max values

That sounds extremely useful. I think sometimes a global min/max might also be useful (when you know your data is within some smaller range), but definitely a local min/max w/ histogram, possibly with some keyboard shortcuts, would make looking at 16 bit data in NG really, really nice.

[jbms]

What do you mean by local vs. global?

[falkben]

oh, i think we were basically talking about the same thing.

i just mean you might want to set a min/max more globally on a channel (esp. if you know the overall channel histogram ahead of time) but then at a particular zoom level (local) it could either automatically scale based on image intensities at that view or could be adjusted by user that way.

[jbms]

Okay, makes sense.

Another thing I'd be interested in your thoughts on is the interaction of multi-channel image sources (i.e. getDataValue(i)) with the histogram and min/max selection.

It would be simplest to just compute a single histogram that combines all channels, and a single min/max that applies to all channels, though having per-channel histograms and ranges could be more useful in some cases.

[falkben]

Discussed with the team here. Consensus was that a single histogram across channels doesn't make much sense for us, as individual channels almost always have very different image intensity profiles.

To be honest, though, we almost always add layers instead of channels (such that it's a 1:1 mapping of layer to channel). However, I'm starting to see that there could be some benefits to having multiple channels within a single layer, as it seems like you could do some neat things across channels with the shaders.

[falkben]

Are num_channels only to be used in the case of RGB datasources? I'm sorry, I may have answered your question thinking that num_channels could be any arbitrary number (and would function like layers).

[jbms]

Neuroglancer volume data sources currently have a numChannels property that can be any integer >= 1. All channels have to be the same data type. The shader for each pixel receives the values for all channels and can use them in any way it likes to compute the output RGBA value. It could do the very simple thing of just mapping channels 0, 1 and 2 to the R, G, and B values of the output pixel, but it could do some arbitrary non-linear function.

In practice channels > 1 is not widely used. It does offer more flexibility for the shader compared to what is possible with just blending between multiple layers, but the downside is that all channels are always downloaded/stored in memory together --- therefore it doesn't work very well to have a large number of channels.

It would be nice if Neuroglancer had an option to combine multiple independent data sources as inputs to a single joint shader, so that the rendering were better decoupled from how the storage format. However, there are some implementation difficulties.

I'm planning to change the way Neuroglancer handles "channels" so that rather than have 3 spatial dimensions plus one "channel" dimension, there will just be an arbitrary number of dimensions, some of which may be marked as usable as "channel" dimensions (because they are not split over multiple chunked or downsampled).

[chrisroat]

Is there some way to access min/max values per channel in the display port? We roll our own shaders like the one written above, and read our (global) min/max values from an attributes file stored alongside our data. I like the idea of having min/max controls, too -- but it gets busy with many channels.

FYI, I use a compromise between multi-channel and multi-layer. We have a natural grouping (9 layers of 4-5 channels), and the layer grouping means I can view multiple UI controls without having to cycle between too many layers.

[jbms]

As an update, I just added a new invlerp UI control type that displays a CDF of the visible data and provides min/max controls for normalization. A given invlerp control always computes the CDF over only a single channel. If you want multiple channels, you can add multiple invlerp controls.

@chrisroat Do I understand correctly that you have stored min/max values as n5 attributes, and are wondering if there is a way to access them from a shader? At the moment, there is not. I think one possibility could be make Neuroglancer aware of min/max attributes and then to use those min/max values as the default range for an invlerp control. Currently the default range is just the full integer range or [0, 1] for float32 data.

[chrisroat]

We have been using precomputed cloudvolumes, and store image statistics (i.e. histogram of values per channel) alongside the volume. We have a library to assemble neuroglancer links, which uses the statistics to create the shader code with 2 per-channel controls: a min slider and a max slider. An example is below.

I'll check out the invlerp UI control!

#uicontrol vec3 color0 color(default="red")
#uicontrol float minimum0 slider(default=110, min=24, max=6082, step=1)
#uicontrol float maximum0 slider(default=179, min=24, max=6082, step=1)
#uicontrol vec3 color1 color(default="green")
#uicontrol float minimum1 slider(default=123, min=0, max=2905, step=1)
#uicontrol float maximum1 slider(default=250, min=0, max=2905, step=1)
#uicontrol vec3 color2 color(default="blue")
#uicontrol float minimum2 slider(default=99, min=0, max=25495, step=1)
#uicontrol float maximum2 slider(default=1900, min=0, max=25495, step=1)
#uicontrol vec3 color3 color(default="yellow")
#uicontrol float minimum3 slider(default=178, min=0, max=5529, step=1)
#uicontrol float maximum3 slider(default=220, min=0, max=5529, step=1)
#uicontrol vec3 color4 color(default="cyan")
#uicontrol float minimum4 slider(default=134, min=0, max=3672, step=1)
#uicontrol float maximum4 slider(default=250, min=0, max=3672, step=1)

void main() {
  vec3 zeros = vec3(0., 0., 0.);
  float norm0 = (float(getDataValue(0).value) - minimum0) / float(maximum0 - minimum0);
  vec3 val0 = color0 * norm0;
  val0 = max(val0, zeros);
  float norm1 = (float(getDataValue(1).value) - minimum1) / float(maximum1 - minimum1);
  vec3 val1 = color1 * norm1;
  val1 = max(val1, zeros);
  float norm2 = (float(getDataValue(2).value) - minimum2) / float(maximum2 - minimum2);
  vec3 val2 = color2 * norm2;
  val2 = max(val2, zeros);
  float norm3 = (float(getDataValue(3).value) - minimum3) / float(maximum3 - minimum3);
  vec3 val3 = color3 * norm3;
  val3 = max(val3, zeros);
  float norm4 = (float(getDataValue(4).value) - minimum4) / float(maximum4 - minimum4);
  vec3 val4 = color4 * norm4;
  val4 = max(val4, zeros);

  emitRGB(val0 + val1 + val2 + val3 + val4);
}