Custom control over segment colors

[stuarteberg]

Currently, users have extraordinarily flexible control over the rendering of image layers, thanks to custom fragment shaders. It would be great to have similar flexibility for segmentation layers. Is that a realistic request? Right now, I have to choose between custom rendering (with image layers) or the selection/deselection UI (segmentation layers). Is it possible to have both?

If full flexibility is not possible in the near term, then the ability to specify a single color for all segments in a layer would be appreciated. This feature request was discussed in a previous thread. In the past I've used the saturation trick, but now I want to create multiple layers with distinct uniform segment colors (for millions of segments).

More brainstorming... Perhaps a simple compromise solution is to support a color table. I have millions of segments, so I can't list each one via a segmentColors mapping. But it would be possible for me to custom-tailor my segment IDs to target a color table of N colors (assuming color = table[segment % N]). Bonus points if the color table can be defined by the user, which would also provide an implicit solution to the "single color" request above. But even a hard-coded color-table would still be useful (assuming it contains a reasonable variety of colors to choose from).

[jbms]

It seems like custom shader support for segmentation layers would be the cleanest solution, though there are some challenges:

• uint64 values are a bit awkward to work with in GLSL, since they aren't natively supported as integer types --- neuroglancer has some utility functions for performing various operations on them, but you can't use the normal operators.

• Currently segment colors are also computed in Javascript for rendering meshes and skeletons, and also for UI components that display segment ids. While for mesh and skeleton coloring it would be possible to move this computation to the GPU (it would be somewhat less efficient, but probably not significantly so), for the UI components would be quite annoying since there is high latency in reading back data from the GPU in WebGL. Instead, it would probably make sense to evaluate the shader from Javascript directly, and there appears to already be a library for doing that: https://github.com/stackgl/glsl-transpiler

[stuarteberg]

uint64 values are a bit awkward to work with in GLSL

My (perhaps controversial) opinion on this topic is that people in our business should simply refrain from using segment IDs above 2^53. That way, we can convert our IDs to/from double with impunity. That can reduce headaches when it comes to cross-language compatibility (e.g. JavaScript, GLSL), and it also avoids other annoying pitfalls. For instance, in pandas, it is all-too-easy to accidentally convert your int columns to double. (And sometimes there aren't succinct workarounds for that issue, even if you're aware of it.)

I'm aware that for some use-cases, people like to co-opt the upper bits to store extra semantics (e.g. chunk location or segment type). It's a natural temptation -- I mean, just think of all those wasted bits! But I guess I'm arguing that those schemes come with hidden costs (in addition to being somewhat brittle), so it's better for people to resist the temptation.

OK, that was a bit of a digression. Here's the question: Would it be possible to add GLSL support with the caveats that the data will be converted to/from double and therefore the behavior for large segment IDs is undefined?

[jbms]

Neuroglancer already defines:

struct uint64_t {
  highp uvec2 value;
};

for representing uint64 values in shader code. I don't think it would make sense to convert to floating point by default, but you could certainly do that yourself in the shader. The easiest thing is if you restrict yourself to 32-bit ids, as then you can just use my_uint64.value[0].

Note that GLSL doesn't support 64-bit floating point, just 32-bit floating point (which can represent 24-bit integers).

While in some cases you can easily avoid using > 53-bit integers, I wouldn't want to impose such a constraint in Neuroglancer.

Another option if Neuroglancer will already contain Javascript code for parsing GLSL is to implement a custom GLSL dialect that supports 64-bit integers natively --- which would basically just amount to rewriting operators and other expressions involving uint64_t into function calls. I worry a bit about creating a custom dialect of GLSL, though. That could also be added later.

[stuarteberg]

Your struct-based solution sounds good to me.

Note that GLSL doesn't support 64-bit floating point, just 32-bit floating point

Whoops, I was looking at the GLSL specification, not GLSL ES. Thanks for the correction.

While in some cases you can easily avoid using > 53-bit integers, I wouldn't want to impose such a constraint in Neuroglancer.

Fair enough -- I know it's a controversial opinion!

As noted above, I don't want to let the perfect solution become the enemy of a decent solution. I bet most people will just be interested in a colortable, anyway (especially if it supports alpha). If that is easier to implement without the full-fledged GLSL solution, then I'd be very pleased to have it. (I have no experience with the neuroglancer code base, so I have no idea what's easy and what's hard...)

[jbms]

Just so I'm clear, is the reason you want to use a segmentation layer rather than an image layer because you also want to display meshes, skeletons and perform object selection?

[stuarteberg]

Right! Object selection and meshes are my priority. In my current use-case, I want to create a layer for cells, and additional layers for organelles within the cells. The cells should be different colors, but the organelles within the cells should be color-coded by type. For instance, the mitochondria layer should be blue (regardless of cell ID) and vesicles should be orange.

[d-v-b]

Chiming in to add that I have the exact same needs as @stuarteberg. Over at www.openorganelle.org we display uint64 segmentations as image layers (which sucks for performance) because we need to chromatically distinguish separate segmentation layers. And if uint64 data is a problem for the shaders, we would be perfectly happy changing our bit depth to something more manageable, since we don't have that many segment IDs to worry about.

[jbms]

I am planning to implement this feature request when I have chance, hopefully soon.

Regarding the performance issue with uint64 image layers, though:

Neuroglancer has two ways to represent volumetric data in GPU memory: uncompressed (which supports all data types), and compressed_segmentation (which supports uint32 and uint64). If the data is originally in compressed_segmentation format, then Neuroglancer always just uses it directly without any transcoding. If the data is in any other format (e.g. raw, jpeg, compressed via an n5 compression method), Neuroglancer first decodes it to uncompressed, then in certain cases may re-encode as compressed_segmentation. Currently, re-encoding to compressed_segmentation only happens when using a segmentation layer, because compressed_segmentation is only effective if the same ids are repeated within local blocks. The assumption is that the data for an image layer is likely to be continuous and compressed_segmentation won't be helpful.

Uncompressed uint64 data takes up a lot of memory, which means only a small region can fit in the cache. That may be the reason for the bad performance that you are observing.

If you export your uint64 label volumes as neuroglancer precomputed with compressed_segmentation encoding, that should be a lot faster since it will greatly reduce the memory required, and will also avoid the need to transcode the data as it is downloaded.

[d-v-b]

is there a timeline for this feature? we would really benefit from this.

[davidackerman]

Hi @jbms, also wondering if this feature is in the works? In many cases it is useful for us to have segmentations where every object in a layer shown as a single color, since in many cases we want to distinguish layers but not necessarily distinguish objects within a layer. For example, we might have a lot of mitochondria, but we would benefit from showing them all as eg "blue" while another layer like plasma membrane would be shown as "red". I know there is the ability to set "segmentColors" in the JSON state, but some of our layers have thousands, or hundreds of thousands, of unique objects. And you mentioned elsewhere that segmentColors should work for a small number of ids. Would a large number of id colors specified via segmentColors hinder performance? Are there existing ways to specify the color?

[austinhoag]

Hi Jeremy, I like the fixed color option you added to Neuroglancer. This helps us out enormously at Princeton. For skeletons, is there anything I can experiment with in the skeleton shader box for customizing the colormap? I'm not sure how to get/set a color there like I do in the image layer rendering tab. I looked through your https://github.com/google/neuroglancer/commit/7f90bb343c98bd073d2b9e08ccbfcf3d6cef07a0 commit and the src/neuroglancer/skeleton/frontend.ts code but it's still not clear to me.

Thanks, Austin

[jbms]

Currently skeleton shaders only have access to the custom properties of the skeleton, not the segment id itself. It would be quite simple to expose it if that is what you want, though.

[austinhoag]

Yes, that is what I was wondering. I think it would be useful for us, so if you could point me to how that is done (maybe with a simple example) we could implement it on our fork if you don't want to add it on yours.

[jbms]

I would be happy to include this change. You would add something like:

builder.addUniform('highp uint', 'uSegmentId', 2);

here:

https://github.com/google/neuroglancer/blob/60c8c036d202cecc7bc09e18b07dc86757bef200/src/neuroglancer/skeleton/frontend.ts#L75

And then make sure to set that uniform when running the shader here (will need to pass in the segment id):

https://github.com/google/neuroglancer/blob/60c8c036d202cecc7bc09e18b07dc86757bef200/src/neuroglancer/skeleton/frontend.ts#L243

Then add some code to be included elsewhere in the shaders to convert uSegmentId into a uint64_t (which is a custom type defined by neuroglancer in shader_lib.ts).