Broader support for AbstractArrays

[timholy]

Volumetric visualization works if you pass an Array, but if you want to visualize a subregion then one might think that view(vol, 100:300, ....) should work. If the failure here isn't deliberate, then we should introduce fallbacks (even if they involve copying data to an Array).

[SimonDanisch]

That should be quite easy to do with my generic convert infrastructure... I will take a look at it!

[timholy]

Mostly I filed this to find out whether this was intentional. If you want, feel free to close this now and address individual issues as they arise.

[SimonDanisch]

Nope, not intentional ;) Just tried to keep down the complexity for a first implementation. By the way I'm playing around with a prototype of doing the whole parameter creation/conversion a whole lot different and more efficient... I should make that effort public to get your input! It basically boils down to giving fields semantic (in form of types), and use those to index the "dictionary" of parameters efficiently and treat the same fields in different types similarly! Since getindex on them will be type stable, I hope to greatly improve performance in my rendering code and make things more predictable for precompilation!

[timholy]

Sounds like it's exactly the same set of aims that the new Images recently achieved. :+1: If you're not intimately familiar with it, the two key contributions were the incorporation of AxisArrays and the creation of a number of view types that allow you to start from basically any type of object and trivially "massage" it into the form that Images wants to work with, without copying any data. Hint: MappedArrays.jl is your friend, because it lets you do the equivalent of reinterpret even on disallowed types.

[timholy]

Oh, and BTW: for orientation purposes I decided to go positional rather than by x and y names. The main reason is that not all images have axes named x and y: for example, with MRI the "proper" axis names are things like R, A, and S. Likewise, for our light sheet microscopy we've adopted names x, l (lowercase L), and s. Hence the new criterion in Images of vertical-first, which is consistent with how matrices are printed at the command line but unfortunately disagrees with "horizontal fastest" so common in other languages.

Note that I'm aware of an orientation issue in NRRD that needs fixing.

[SimonDanisch]

@timholy I finally got around to turn my prototype into a package: FieldTraits I want to make this kind of composed type the new underpinning of GLVisualize ( or rather GLVisualize) Let me know what you think of the design!

[timholy]

Before I dive into the code, can you briefly summarize for me the problem you're trying to solve? I think all the issue of naming axes are well-handled by AxisArrays, and all the issues of permuted order are well-handled by PermutedDimsArrays. They both have systematic support through the standard type system for many operations.

[SimonDanisch]

Sorry, this wasn't very clear. This is not about Arrays and Images, this is about sharing documentation, conversion behaviour and attributes across generic visualization types. I simply chose an Image, because it's the simplest primitive in GLVisualize and it was straight forward to implement the conversion behavior previously defined in image_like.jl .

of naming axes are well-handled by AxisArrays, and all the issues of permuted order are well-handled by PermutedDimsArrays.

In GLVisualize, my task is to extract all useful information out of a type a user gives me and transform into something a graphics backend can work with. This means, when they give me an AxisArray, I need to update certain attributes of my visualization. When they give me a plain Array, I need to fill in defaults, that the Array type doesn't contain, but which I need to display the image.

Covering all these conversion and default creation tasks can be elegantly solved with FieldTraits. It becomes especially important, when I offer different backends, which have almost the same behavior (in regard of creating defaults etc) but might slightly differ from field to field (E.g. I expect the behavior of the Ranges field to be shared across all Backends, while I will need to define special conversions for OpenGL and the ImageData field, since I want to convert it to Texture{RGB{Float32}, 2}) Note, that a few other visualization types contain an ImageData field, for which I don't want to duplicate that logic (e.g. a textured mesh).

[SimonDanisch]

This is basically the answer for my need, that GLVisualize is 90% about documenting, converting, handling wrong formats, etc for all different kind of visualization attributes, which are usually shared between a lot of visualization types, but might have slightly different behaviour depending on the parent type that contains the field/attribute. This problems becomes a lot worse now that I want to offer different backends, since backends mostly duplicate behaviour but also might differ greatly depending for some field.

[timholy]

OK, that makes more sense. In the modern JuliaImages, we provide views that make it easy to work with both the user-provided format and the Images-preferred format. I think that makes sense for something like an image-processing suite, where you're likely to have a whole chain of calls generated by the user and consequently it's nice to have the intermediate "standardized" format so it can be reused. For a visualization suite, in a sense once you "slurp in" the user's input, then it becomes opaque to the user, so I guess there's less motivation to expose convenient "standardization" views.

I probably won't have time to check it out today, but I'll take a look ASAP.

[SimonDanisch]

Great, I'm looking forward to your feedback :) Bug me if there's anything that doesn't make sense! I will try to improve the examples and README as well.

[SimonDanisch]

I wrote up some more about my design and how Visualize will interact with FieldTraits in the README Hope it makes a bit more sense! Btw, my pure Julia backend comes along very nicely :) See my pure julia rasterizer and a simple "opengl" program written with it: source Result (still has precision problems):

The awesome thing is, that we can transpile this to OpenGL, being able to freely mix Julia and OpenGL :)

So it will be possible to e.g. do this:

volume = rand(Float32, 128, 128, 128)
visualize(volume, pathtracer = (ray, volume) -> begin  # do path tracing in a julia callback on the gpu!! :-O
     ...
end)