Should Gray be <: Number?

[fghzxm]

Since Gray{T} supports whatever numeric operations T supports, I think it is reasonable to make Gray <: Number so that code that accept Numbers also work with Gray. I personally encountered this problem when I tried to do DSP.conv between two grayscale images: since DSP.conv only works with arrays with eltype <: Number, I had to convert my arrays back and forth between Array{Gray{T}, 2} and Array{T, 2}.

[johnnychen94]

The path we're taking in Images.jl is to treat each pixel as a complete unit of an array element, so that Gray and RGB can be treated more or less in the same way in the high-level API. As a consequence of this design, for example, the size of an RGB image with height 16 and width 16 is (16, 16) instead of (16, 16, 3); the size doesn't change even if we convert it into a gray image using Gray.(rgbimg).

Gray and RGB are both Colorant, and Colorant, in general, doesn't follow the rules of scalar or vector operation. Hence we can't make Gray <: Number here; doing it would require a re-design of the core system of Images. Ideally, we want to avoid the use of a numeric array in favor of a colorant array. This isn't perfect yet, but as time goes on, we could work together to cover more use cases so that we could fire up without the use of the numeric array.

A common solution taken in Images to provide such support is to provide a thin wrapper for that. For example, ImageDistances is a wrapper of Distances with Colorant support, and ImageAxes is a wrapper of AxisArrays, too. If you find DSP useful in your daily work, you could provide a wrapper for that as well.

In case you don't know, channelview(x) would be helpful in this case, too.

Edit: I'm transferring this to JuliaImages.github.io as I believe it's more of a documentation issue.

[timholy]

I think DSP should look at dropping the Number requirement, presumably anything that supports + and * should be good enough? Or does it use FFTs?

Note ImageFiltering supports correlation and colors.

[galenlynch]

DSP does use FFTs for its convolution at the moment, though it would be great to add a direct convolution method which just uses * and +, and would allow for arrays of non-numbers. Of course I could look this up myself, but out of curiosity does ImageFiltering support FFT convolution of Colorant data? Last time I looked at the convolution options on ImageFiltering, I seem to remember being able to select the algorithm, and also the hardware used to execute the convolution (impressive if that's right!).

[timholy]

Since it's designed to work for images, it looks specifically for colors and uses channelview on arrays of colors. This is only needed for FFTs, for FIR implementations ::Real * ::Color is well-defined.

[johnnychen94]

Reposted from slack (I completely forgot about this issue):

I’m starting to think about what Number is. Is it scalar-like stuff in the sense of broadcasting and happens to share a lot of properties of numbers? The fact that RGB is a “color vector space” is pretty much the same case as Complex. Maybe it’s a good idea to introduce a calculation-only wrapper type

struct ColorVector{CT<:Colorant} <: Number
    c::CT
end

and associated conversion from/to Colorant for computation purposes. This can explicitly split types into two-stage: storage, and computation. Will this make it easier to write generic codes? One advantage I can think of is that many packages may not need to introduce the ImageCore dependency only for the color types, and glue package such as ImageDistances doesn’t need to exist.

The reason I’m starting to think about this is that recently I’ve been developing a package to represent a Galois field number. My first thought is to make it a GaloisFieldNumber <: Number, it works quite well. But since this generic algebraic field number is so special (e.g., + is defined as xor), it quickly turns out that many Julia codes that declare Number doesn’t really assume it’s a Number, but instead something like Real and possibly Complex. This makes me think about what Number really is and what assumption we can make about it, which is why I’m thinking about the possibility of introducing a Number type color.

[rafaqz]

I've actually been thinking about this too. It would be good to make the color vector space an explicit wrapper rather than by just loading the ColorVectorSpace.jl package if you get an error.

We could also have better errors in the ColorVector constructor for why e.g. HSL doesn't work this way.

There is the code in https://github.com/SimonDanisch/AbstractNumbers.jl for an overview of what "being a number is" in Julia, but it can be frustrating how vaguely it is defined.

[timholy]

The boundaries between abstract principles are always a bit uncertain.

Stating one obvious point, we can't make Gray <: Number without having all Colorant <: Number since Gray <: Colorant.