Closed aaronfranke closed 2 years ago
scalablecory
commented
4 years ago How about System.Windows.Media.Color, which uses floats?
If we were to introduce a new color type, I would like to consider ways to represent linear RGB, LCHab, LSHuv, etc.
thargol1
commented
4 years ago @aaronfranke in general this is called 32-bit color, so would the name Color32 not be a better name? This would be similar to Int32.
scalablecory
commented
4 years ago I'd say it's more frequent to see colors referred to by their component size when using them in code.
Turnerj
commented
4 years ago This might be answered elsewhere but why the System.Numerics namespace? Not that I have a better suggestion, just more curious to the choice.
aaronfranke
commented
4 years ago @Turnerj Mostly because I don't know where else to put it. System.Numerics is also where the vector types are, so it seems fitting since they are similar in structure (but not similar in purpose).
tannergooding
commented
4 years ago I'm not convinced adding a new Color type is the solution here. Everyone will have their own ideas on the correct shape, layout, fields exposed, etc.
It might, however, be beneficial to expose some of the common algorithms and have them operate over Vector4. DirectX Math provides several functions like that here: https://docs.microsoft.com/en-us/windows/win32/dxmath/ovw-xnamath-reference-functions-color As do other similar libraries like GLM: https://glm.g-truc.net/0.9.0/api/a00143.html, https://glm.g-truc.net/0.9.0/api/a00144.html, https://glm.g-truc.net/0.9.0/api/a00145.html
They would then also logically fit into the System.Numerics namespace and find parity with several of the other types here that are designed for use in games/multimedia based applications.
tannergooding
commented
4 years ago I'd also be interested in what others, like @JimBobSquarePants or @rickbrew have to say, as they both maintain large/popular apps/libraries in this space.
JimBobSquarePants
commented
4 years ago This reminds of the old XKCD comic about competing standards.
There's a reason each of the mentioned libraries have their own color type and it's because their specific needs require them.
There's simply too much to consider in the graphics space to allow the design of a small collection of types to cover those needs and since that space requires expert knowledge to develop specific common algorithms, most developers won't want to write them and will choose, instead, to adopt a complete ecosystem within which to operate.
This requires finely tuned color structs which cater for things like rgb working spaces, bit depth, and packing order.
I have to be honest and say that I cannot see an obvious benefit in adding these types. The major graphics players in the .NET ecosystem all have their own types already and this will not provide unification among them.
Turnerj
commented
4 years ago While I do agree with @JimBobSquarePants that it won't be a perfect solution for anything terribly graphics intensive and they would all likely roll their own for various reasons, I do think a basic Color type is worth having.
Sometimes, I just want to refer to or accept a colour in a method without bringing in other dependencies or rolling my own type - I want something simple and generally interoperable.
While System.Drawing.Color does exist and otherwise would fit this purpose, it has potential legacy baggage and some weird quirks (see https://github.com/dotnet/runtime/issues/30208).
From my point of view, either a new Color type would work or (preferably) fix some of the underlying quirks of System.Drawing.Color.
tannergooding
commented
4 years ago Thanks for the feedback @JimBobSquarePants. It's about what I expected and is generally inline with what I was feelingπ
Just for clarification, however, would you feel the same way about having some "color" functions that operate on Vector4 exposed under the System.Numerics namespace (rather than exposing new types)?
The idea being that each library has their own types, but many of the core operations are standardized (hence the support from both OpenGL and DirectX) and often have vector, GPU, or shader acceleration available. So it would be possible for image libraries to reuse some of these core functions without needing to remove or change their existing types.
https://github.com/dotnet/runtime/issues/32418#issuecomment-595397329 has links to the respective API surfaces exposed and would give an example of the kind of functions that would be considered.
aaronfranke
commented
4 years ago I have no doubt that people will have their own use cases for Color types, and some people may choose to use their own (and it's great that C# allows this). However, I don't think that this means we shouldn't have a general-purpose Color type inside of .NET.
Having this is a great idea because it would be useful for anyone who doesn't want to build their own Color type or import a library.
A Color type in .NET Core would also serve as a common interchange format, so that libraries wouldn't have to depend on each other by referencing each other's color types, or doing something like using four floats, or a Tuple of floats as an interchange format. I had a similar discussion with developers before when I proposed adding (Value)Tuple operators to vectors, and they said it would just be better for System.Numerics.Vectors to be always included in .NET Core and then libraries can reference those types and use them for conversion.
rickbrew
commented
4 years ago I don't think it's possible to have a "general purpose" color class/struct. No matter what you implement, there will always be too much "what about this other use case?" going on.
First, I'd highly recommend studying how Windows Imaging Component implements pixel formats : https://docs.microsoft.com/en-us/windows/win32/wic/-wic-codec-native-pixel-formats
Also, it's useful to be precise and careful with the vocabulary around colors and pixels (and pixel formats). They get conflated a lot, they are related and definitely overlap, but they are also not the exact same thing. However, it is difficult to fully separate the concepts, especially because of how they are colloquially blurred together, so you can't actually get too pedantic. As an example, System.Drawing's Color is definitely a color and not a pixel. It is not suitable to create a 2D array of them and call it a "bitmap" (just look at its fields).
I think I agree with @JimBobSquarePants on this. I think that the idea of having of a general purpose Color type is a wonderful utopian idea, and would be useful, but I'm not sure how to achieve that given the incredible complexity in this space. Even though I've been working with computer graphics for over 20 years, I'm still consistently baffled by new things I learn about colors every year.
I think a bottoms-up approach has worked well in the past. That is, start with a graphics library (e.g. GDI+/System.Drawing, Direct2D, ...) and define -- within the library itself -- the pixel formats that it can natively work with. I don't think the top-down approach being proposed here is the right place to start.
What isn't working well right now is that every graphics library has all its own definitions of these pixel formats, and there's no standardized way to map between them, how to map to vector/GPU types and back, or even how to map to simpler definitions that are also useful (e.g. enum Colors { Blue, Red, White, Black, ... }) etc. WIC solves some of this on its own with the concept of format converters, as well as color spaces and color transforms aka color management (https://docs.microsoft.com/en-us/windows/win32/wic/-wic-colormanagement). It works pretty well, but as you can see it gets complicated fast! (and, WIC is extremely unapologetic with its complexity) Also, WIC handles "bulk" conversions (full rows or whole bitmaps). It's still useful to have an efficient way to convert 1 pixel/color at a time, which is easier to achieve in .NET or C++ than it is in COM. How do you map a 32-bit-per-pixel BGRA sRGB color over to 128-bit-per-pixel RGBA HDR? Β―\(γ)/Β― I dunno, but having that functionality would be awesome.
I disagree that the number suffix usually denotes the bits per component. In my code base it represents the total size of the color, e.g. ColorBgra32 is a 32-bit color with B, G, R, and A integer components that are 8-bits each. ColorRgba128F is 128-bit color with R, G, B, and A floating point components (same as what Direct2D uses). Unfortunately the color space is unspecified, but sRGB is the de facto standard right now. Also, if the number denotes bits-per-component, how do you disambiguate between colors that have the same bits-per-component but a different number of components? In Paint.NET, I have e.g. ColorBgra32 and ColorAlpha8. Both have 8-bits-per-component, but 4 or 1 components (respectively). The latter is used for alpha masking, and is a very important type for graphics work. As you move towards lower-level graphics libraries like Direct3D, just saying Color8 would be waaaaaaaay too ambiguous.
I think this space will all only get more complicated as time goes on (see also: HDR, good luck understanding that space! π), so I don't think there's any safety in finishing a project like this and thinking it would be the be-all-end-all result. It would need revision every few years, if not more often.
I think there's definitely value in having standardized Color types that map to Vector and GPU types. I'd love if there was some kind of implicit reinterpret_cast-style ability to do custom mappings here.
In Paint.NET, I've settled on a collection of Color____ primitives that mirror and map to those that are provided by Direct2D and Windows Imaging Component (WIC). Each struct then implements IPixelInfo so that it can describe itself and be used in generic and format-agnostic code. This is analogous to WIC's IWICPixelFormatInfo interface:
public interface IPixelInfo
{
int BitsPerPixel { get; }
PixelFormat PixelFormat { get; } // directly maps to WICPixelFormatGUID
}
// "natural" may not be the best term, but it just means the size is measured in bytes, not bits.
// in other words, simple pointer arithmetic can be used for all inter-pixel navigation
// so this doesn't cover 4-bits-per-pixel or some other esoteric formats that are 10 or 30 bits per pixel or whatever (if those even exist)
public interface INaturalPixelInfo : IPixelInfo
{
int BytesPerPixel { get; }
}
// makes a lot of generic coding a lot easier
// the IUnsafeElementAccessor thing is largely unimportant now since the release of C# 7.x and System.Runtime.CompilerServices.Unsafe.*
public interface INaturalPixelInfo<TPixel> : INaturalPixelInfo, IUnsafeElementAccessor<TPixel>
where TPixel : struct, INaturalPixelInfo<TPixel>
{
}
public struct ColorBgra32 : INaturalPixelInfo<ColorBgra32> { public byte B, G, R, A; ... }Sometimes, I just want to refer to or accept a colour in a method without bringing in other dependencies or rolling my own type - I want something simple and generally interoperable. @turnerj
tl;dr from my novel above, "simple" and "generally interoperable" are extreme oxymorons in this space π
rickbrew
commented
4 years ago A Color type in .NET Core would also serve as a common interchange format... @aaronfranke
I'm not convinced a "common interchange format" can be achieved in this space. It can certainly be done for a very narrow subset of functionality and use cases, at which point it's definitely not general.
aaronfranke
commented
4 years ago Of course we can discuss what "simple" or "general-purpose" or "generally interoperable" mean, that's why this discussion exists, but I disagree with the notion of "I don't think it's possible to have a "general purpose" color class/struct", the message I'm getting from this is basically 'this is so incredibly complicated that we shouldn't even try' as if there aren't advantages to having something roughly general purpose in .NET.
How do you map a 32-bit-per-pixel BGRA sRGB color over to 128-bit-per-pixel RGBA HDR?
That's the purpose of the byte constructor, you provide it with a byte for each channel and the constructor maps it such that the range [0, 255] becomes [0, 1]. If you are expecting some other range, you can modify the resulting Color as needed (or build your own method).
I'm not opposed to renaming Color8 (or really, even dropping it, I mostly care about the 32 bits per channel floating-point Color myself since it is the most flexible format).
It can certainly be done for a very narrow subset of functionality and use cases, at which point it's definitely not general.
We can just add more functionality if we determine that a given use case is common enough to warrant such functionality. I'm not saying we will make something and never touch it again, calling it done and good for general use cases, we figure out general use cases by user demand and we always have the option of adding more functionality later.
saucecontrol
commented
4 years ago I laid out most of my concerns in https://github.com/dotnet/runtime/issues/14825#issuecomment-497825808, but I'll add a few replies to the comments here...
@rickbrew It's good to see someone else aboard the WIC train. WIC has a hell of a learning curve, but the more time I spend with it, the more I like the way it handles extensibility. The WIC pixel format definitions are a great example of what it takes to define those in a truly agnostic way.
@aaronfranke If you take away the methods that rely on a specific interpretation of the color data so that your 128bit-per-pixel format is truly generic, how does it differ from (or improve on) Vector4? Vector4's X, Y, Z, and W properties can be easily interpreted as B, G, R, A or A, R, G, B or C, M, Y, K or whatever other format you like. Vector4 also comes with the benefit of being recognized as a JIT intrinsic so it can be passed in a SIMD register automagically and has lots of useful math already defined.
@Turnerj have you looked at System.Windows.Media.Color? It solves a lot of the design problems with System.Drawing.Color if you need something to represent an actual color rather than a pixel value.
@tannergooding The DirectXMath color utililty functions could be pretty useful if they were built as extensions that operated on Vector4. I fear, however, that they will become increasingly irrelevant over time because it's no longer safe to assume that all colors live in either the Rec.709 or sRGB color spaces (or the linear space with the same whitepoint and primaries). Wider color gamuts and higher dynamic range will become the norm over the next few years, so by the time those additions would be useable by any number of developers, they'd be on their way to obsolescence.
tannergooding
commented
4 years ago they'd be on their way to obsolescence
I see your point to an extent, but that is also why those libraries continue adding new functions each release (for example XMColorRGBToYUV_HD and XMColorYUVToRGB_HD are relatively new). The goal of DirectX Math and GLM is to provide the most core functions needed/used by games and multimedia based applications (and so they cover the core standards like ITU-R BT.709, ITU-R BT.601/CCIR 601, CIE XYZ, and IEC 61966-2-1:1999).
However, the functions are also well-defined and MIT licensed (at least for DirectX Math) and are fairly trivial to implement using the existing Vector4 surface area (and the new functions which allow converting Vector4 to/from Vector128<float>), so it also isn't something that is "critical" or that would block anyone from providing their own library for these things if needed/desired.
rickbrew
commented
4 years ago Yeah, I'll walk back from "it's not possible", but I still think it's ultra important to be very specific here. What you want is an ultra specific definition that is capable of being employed casually by folks who don't need or want to dive in and read a 500 page book called "Colors and Pixels: Volume I" π
I think starting with what WIC and Paint.NET do would be a good place to start. The primitive/low-level "types" here should be pixel formats, color spaces, and conversions between them. Any color/pixel struct would implement interfaces that provide information about the pixel format, memory layout, color space, etc. This strategy permits precision and also leaves a lot of room for expansion later.
Once you have a bunch of structs you can define implicit and explicit casting operators, as well as ways for converting between them. For instance, ColorBgra32 should be implicitly castable to ColorRgba64 *, but going the other way would require an explicit cast. Converting to something like (for example) ColorHsv96F (HSV, 32-bits per component floating point) would need a converter method or class, especially since you'd be losing the alpha channel (this would technically be a blending operation done "over" an explicit, or perhaps an implicit default, background color), and also be subject to rounding/clipping (it's not always a reversible conversion).
@saucecontrol re: S.W.M.Color, I think this is similar to Direct2D's color struct with some WIC mixed in. WIC, WPF, and Direct2D all share a common heritage and have a lot of overlap. D2D's color struct doesn't include the color context information, but it is still mostly used by itself rather than packed tigntly together into a bitmap. For example, you use a single color value to instantiate a ID2D1SolidColorBrush, or for gradient stops, etc.
@saucecontrol 's notes on how WPF's color struct includes the color space is an important one. To fully define a "color" you need to specify the pixel values, the pixel format, the color space, etc. To display the color correctly you need to match it up with the monitor's color profile. An important thing to note here is that storing this extra information is in-line with the goal of having a color/pixel struct that is "general purpose", but in conflict with having one that maps to vector/GPU types. Unless you make that information part of the type system like I do in Paint.NET (see my comment above), at which point you need one struct per variation.
* footnote here ... on Windows the de-facto 32-bit color is BGRA, while 64-bit is RGBA. For some reason I'm not currently privy to. Whatever :)
tannergooding
commented
4 years ago I think there's definitely value in having standardized Color types that map to Vector and GPU types. I'd love if there was some kind of implicit reinterpret_cast-style ability to do custom mappings here.
@rickbrew, we currently have ref TTo System.Runtime.CompilerServices.Unsafe.As<TFrom, TTo>(ref TFrom value) and you can do pointer casts as well. So *(Vector4*)color and *(Vector128<float>*)color are both valid, assuming color is 128-bits in length.
The codegen, however, isn't always perfect today and is something we are tracking more generally (we do have zero cost conversion between Vector4 and Vector128<float> for .NET 5, however).
If you have a more explicit ask or need, feel free to open an issue and we can start the discussion on it π
saucecontrol
commented
4 years ago those libraries continue adding new functions each release
Ah, that makes sense. I was kind of impressed they covered Rec.709 in addition to the usual sRGB, but then they're missing Rec.2020 and Display P3 which are definitely bordering on mainstream at this point.
I could certainly see a demand for a common library of accelerated conversions between those standards-derived formats. I'm not sure the BCL is the place for them, but that would be a good library to have.
rickbrew
commented
4 years ago @tannergooding Yeah that's something I don't actually have any specific needs for at this time. I've not yet converted over to .NET Core, and I've so far been doing vectorization in native code. At some point I'll jump into Vector4 :)
Turnerj
commented
4 years ago Sometimes, I just want to refer to or accept a colour in a method without bringing in other dependencies or rolling my own type - I want something simple and generally interoperable. @Turnerj
tl;dr from my novel above, "simple" and "generally interoperable" are extreme oxymorons in this space π @rickbrew
From my perspective of "simple", I'm just wanting the lowest common denominator. Like you've described, there are so many details when it comes to colour that a general Color type that could do everything is a utopian future. I just want something to pass around without inventing my own types or taking on dependencies.
This is likely based on my view of .NET: I see the runtime bringing all the basic types and functionality to the table that people might reasonably need. These effectively act as a lowest common denominator - never meant to do everything but do just enough in a lot of cases without a developer needing to re-invent the wheel.
When you get to the "I need to handle HDR colours" (etc) point, that's when you start bringing in specialised libraries where people have built that functionality. Kinda like if I wanted to do image processing, I could do some stuff with System.Drawing but if I'm too restricted by it or have some other requirement it can't fulfill, I'd bring in ImageSharp.
@Turnerj have you looked at System.Windows.Media.Color? It solves a lot of the design problems with System.Drawing.Color if you need something to represent an actual color rather than a pixel value. @saucecontrol
My experience is fairly limited with System.Windows.Media.Color and while I do know it has .NET Core 3 support, isn't it somewhat intrinsically tied to WPF? I don't know what the future looks like for WPF with .NET on Linux - I mean, I might still be able to access the type but if WPF is a Windows-only technology, I could imagine very few API surfaces by third parties taking in that type.
rickbrew
commented
4 years ago My experience is fairly limited with
System.Windows.Media.Colorand while I do know it has .NET Core 3 support, isn't it somewhat intrinsically tied to WPF?
I agree, I don't think you can standardize on System.Windows.Media.Color specifically, for the dependency and platform reasons you give. It's in PresentationCore which is tied to Win32. But it's probably a good starting point of functionality for steering this proposal.
That struct is pretty heavy weight, containing a ColorContext (reference type), bool, float[], and both a MILColor (4x byte) and MILColorF (4x float) structs. So, maybe around 180 bytes on its own plus whatever that float[] takes up (seems to not always be used). Definitely not usable for high performance, but should be fine in many other contexts.
Conceptually, S.W.M.Color looks like a discriminated union between a 32-bit sRGB BGRA (8-bits-per-component) and a 128-bit scRGB BGRA (32-bits-per-component, floating point) color value.
rickbrew
commented
4 years ago I don't know what the future looks like for WPF with .NET on Linux
My guess is, don't hold your breath. One of the blockers for porting WPF anywhere other than Win32/x86/x64 is the C++/CLI compiler. It's stuck on Win32/x86/x64, and I have no idea when or if we'll see arm, arm64, or non-Win32 support there. It's a blocker for Paint.NET as well, I have a lot of native glue code in C++/CLI.
WindowsBase may have a chance of being ported elsewhere, I'm not sure what Win32 dependencies it has. But that just gives you, what, the data binding engine? Nothing useful for directly building a UI.
saucecontrol
commented
4 years ago Yes, x-plat is a definite sticking point with S.W.M.Color. But it does serve as a reasonable model for what a generic color-defining struct would be. Its other major downside is that it only supports RGB(A) colors, which means you'd need something different to work with CMYK, LAB, YUV, or any other color model. Those could both be solved by defining yet another color type or types, but that gets out of hand quickly if we're talking about a lowest common denominator type.
Adding a third incomplete RGBA-only color definition to augment the two existing ones seems like a mistake. Maybe it's possible to break the dependencies on Windows to create a WPF equivalent of System.Drawing.Primitives that are managed-only and cross platform. That would still have shortcomings, but at least it wouldn't pollute the API any further.
And as @rickbrew pointed out, that concern is orthogonal to the need for something that represents pixel values that you would store or process in bulk, because it would be far too heavy.
For at least 20 years, I've heard developers saying "I just want to be able to say 'Red' and have it mean 'Red'". The reason that problem hasn't been solved in all that time is because it literally can't be. The closest we ever came was sRGB and the W3C's adoption of sRGB as the one true color space. That's about to go back out the window as display technology has passed the limits of sRGB. How do you express the Red that's the reddest Red my display can show, when Red is already defined as 255,0,0? (See the first image @ https://webkit.org/blog-files/color-gamut/comparison.html if you're using a better-than-sRGB display to see what I mean)
aaronfranke
commented
4 years ago when Red is already defined as 255,0,0?
Semantically speaking, with floating point color math, 1 means 100% intensity, so 1,0,0 will always mean fully red. Floats allow you to perform whatever operations you wish, including representing values outside of the normal 0-1 range, but at some point you still have to convert to bits on a display, at which point the only sensible option is to scale down to the 0-1 range, where 1 is full intensity.
So, on a 10 bits per channel display, while the maximum integer intensity would be 1023, and therefore 255 would not be the brightest anymore, the maximum float intensity is still 1.
saucecontrol
commented
4 years ago You're talking about operating on a different scale, not working in a different color space. What if Red in my color space is actually 0.8,0,0.05 instead of 1,0,0? There's no way to convert without a color profile (or as WIC and System.Windows.Media call it, a ColorContext).
For a more extreme example, open this image in MSPaint or any other non-color-managed application:
rickbrew
commented
4 years ago And if you want to blow your mind a few more times, this series of educational articles about color management from 13 years ago is great: http://regex.info/blog/photo-tech/color-spaces-page0
Reading that is what convinced me to remove automatic conversion to sRGB when opening images in Paint.NET, something I almost shipped in v4.2 (July 2019). It's just way too complicated and I would've made things worse.
Yum, bluesberries
rickbrew
commented
4 years ago Semantically speaking, with floating point color math,
1means 100% intensity
100% intensity relative to what though? The camera's sensors, my monitor's pixels, my eyeballs?
There's no such thing as 100% brightness, you can always add more photons/energy to light
saucecontrol
commented
4 years ago educational articles about color management from 13 years ago
Ha, that's the exact series that led me to create some of my own wacky profiles for testing.
And yeah, since the spoiler is there, the raspberries (or bluesberries) are encoded in that image as ~ 55,70,170, which how I define Red in my wacky colorspace.
Of course, that's not terribly realistic, but "more red than the reddest red you can imagine" does not simply mean pumping up the red channel from sRGB to greater than 100%
antonfirsov
commented
4 years ago Defining a generic, universal type to represent colors in a way that is agnostic from all perspectives, and at the same time capable for meaningful operations on instances in a convenient and performant manner is pretty much impossible.
In ImageSharp we have overcome this issue by defining a Color type that keeps it's state completely private.
This Color type is pretty much useless for anything else than passing color information from A to B in an agnostic way. When some algorithm actually needs to extract and use that information, it should convert Color instances to pixel instances of a specific representation.
Any other solution would - by design - suffer from the issues already well explained by @rickbrew's comments.
juliusfriedman
commented
4 years ago @tannergooding @antonfirsov @rickbrew @aaronfranke et al
Just wanted to chime in with what I had going on a while ago FWIW.
I originally started with something I copied from j-codec @ https://github.com/juliusfriedman/net7mma_core/blob/master/Codecs/Image/ColorSpace.cs
That got rather limited IMHO and I found out was based on libav to some extent which is somewhat surprising... so I dug a little deeper and found out that libav also had some issues with certain things e.g. very weird non real color formats that I could imagine but not see how to get to work easily in libav. (mostly custom alpha formats I think)
I ended up with:
ImageFormat: which is still more wordy than I would like but seems to cover ANY format I could think of.
You have to define your transformation ONLY on the source format as the intermediate is RGB right now but that can changed in the implementation of the Transformations if needed as required by the implementation.
The top level classes are: Media Format Which has the MediaType, Size, Media Components [Which are basically the channels of data because this is supposed to work for Audio and Video in the same way] and finally it's MediaBuffer which is your Memory.
You can then get to Audio, Image or Video specific things:
https://github.com/juliusfriedman/net7mma_core/tree/master/Codecs/Image https://github.com/juliusfriedman/net7mma_core/tree/master/Codecs/Audio https://github.com/juliusfriedman/net7mma_core/tree/master/Codecs/Video
MPEG is also implemented to some degree (MPS, TS PES, Etc)
Hopefully it can be useful if nothing else just to serve as a reference from a different point of view where as in this library all colors are thought of as parts of media component which defines how to turn it into either RGB, HSL, or any other color space required by the application with little effort.
Ideally applications should stay in their own color spaces and not need to transform and if they do then typically that is where acceleration is required to reduce the jitter in the encoding / decoding (transcoding) process and all of the other fun complexities as partially shown above.
I think having something like Color is obvious but then to have it's components and be able to drill down into the raw data is very important. I think we should also consider audio and image as just types of media and let the user decide how he wants to view it (think for a [color]blind or deaf person or otherwise disabled person, they are going to require special trans-coding anyway)
E.g. the input and output could be anything you just have to be able to define them, which is easy using the MediaComponents which tell you what they are for and how big they are (each component and how many there are [how to handle odd components etc])
i.e. I am sure we all know you could literally take audio and make it an image in a certain bit depth and color space and you would definitely see something (potentially the environment or device from where the audio was sampled if you performed the right inverse quantization and filtering but more likely what is known as noise when unrefined.)
I think in my implementation in the end a Generic class would be able to be provided for any Transformation which would be supplied with an optional ColorSpace argument to do the underlying conversion given the information it knows about the size of the components and how most efficiently to ready them.
The problem as stated above is that then you need to transform the Red in one space to the Red in another space and that is what the Transformation does so as long as someone has a point of view on a Red it can be converted to another space but might not look the same depending on that space and the values it supports which is where trans-coding is required. (Transformation)
aaronfranke
commented
4 years ago By the way, even if there is no color type implemented for .NET Core, we should still deprecate System.Drawing.Color.
tannergooding
commented
4 years ago What would be the reason for deprecating System.Drawing.Color? It is still used for its intended purpose, with System.Drawing; which itself is based on GDI+ and is actively supported on both Unix and Windows.
nxrighthere
commented
4 years ago There's no point to deprecate it without a replacement. It makes sense to update/improve it, but not just leaving behind.
It's used extensively in game engines for basic colors where 8 bits of precision per channel is acceptable such as console messages. For something advanced, more precision is required for sure. For instance, I've implemented a custom LinearColor structure for the game framework for shader materials and so on. It would be nice to have something similar out of the box in .NET.
aaronfranke
commented
4 years ago Then why not replace it with the proposed Color8 type, and add the proposed Color type? They work for the existing use cases of System.Drawing.Color, and are flexible enough to be used for more use cases.
The argument against this is basically that doesn't work for every single use case, but it doesn't need to. We don't need to support many color spaces and such, we can just treat 1.0f in the proposed Color as 255 in System.Drawing.Color.
To me it doesn't make sense to argue against a .NET Color type because it doesn't cover all use cases, and then also keep System.Drawing.Color which covers even fewer use cases.
juliusfriedman
commented
4 years ago Why not use something like I have proposed , it is extensible enough that I could also add the same format as @nxrighthere posted and would that be sufficient? It will only benefit users of shaders on GPUS so I suspect not...
My implementation supports audio and video in the same framework and is able to be SIMD optimized, it's basically a modernized take on what libav did.
I am not saying @JimBobSquarePants's library is bad I just don't think it will ever support Audio easily without a bit of work to add that underlying intermedia which would be able to marshal the types (if safety was even a concern at this point)...
There are also proposals for sound / media and there previously was also sound support in full framework albeit limited, it would work well enough to achieve what you needed without interop most of the time.
Why just not have the users vote on their favorite graphic / media API and then from there just add a way to get the raw buffer if not already exposed and then we can do that, it's what you guys basically did for libuv right?
Heck for all that we might as well just ask libav to be able to link against them and offer a build option with MEDIA_SUPPORT=.... to the build process.
@aaronfranke, there are lots of people using GDI plus and those users will not benefit from optimizations in the Color code no matter how advanced because they call into GDI which needs things in a specific way in order to do it's job. Changing Color on them will do no good.
I saw we deprecate all of system.drawing and system.media and come up with a complete replacement or fix it.
tannergooding
commented
4 years ago What we already have covers the basic use-case, which is accessing and interpreting colors in a fairly typical format (8-bits per pixel, 3 color channels + alpha channel). We also have System.Numerics.Vector4 which can be the basis of a hardware accelerated floating-point format (32-bits per pixel, 3 color channels + alpha channel) and covers a good number of additional scenarios, including conversion between formats.
Anything additional starts getting into very specialized territory that likely isn't a good fit for the actual core library and that is better served by more specialized libraries, like ImageSharp, Windows Imaging Component, or others available projects. There is some additional functionality that could be exposed for these basic types we already have that would make them feel "more complete", even for the somewhat limited scenarios they support (basic image manipulation and hardware accelerated graphics).
At the end of the day, what we ship in box will never include stuff to satisfy everybody and dev's will need to create or depend on additional third party dependencies in some scenarios. There are benefits and drawbacks to this, just like there are benefits and drawbacks to tying your audio library and video library together, etc. Having smaller more actionable proposals to extend the existing stuff we expose is likely more actionable and while we can have larger discussions about more complete replacements, that is also a much bigger ask and has to be weighed against existing libraries that are likely doing it better (and likely will always do it better).
juliusfriedman
commented
4 years ago What we already have covers the basic use-case, which is accessing and interpreting colors in a fairly typical format (8-bits per pixel, 3 color channels + alpha channel). We also have
System.Numerics.Vector4which can be the basis of a hardware accelerated floating-point format (32-bits per pixel, 3 color channels + alpha channel) and covers a good number of additional scenarios, including conversion between formats.Anything additional starts getting into very specialized territory that likely isn't a good fit for the actual core library and that is better served by more specialized libraries, like ImageSharp, Windows Imaging Component, or others available projects. There is some additional functionality that could be exposed for these basic types we already have that would make them feel "more complete", even for the somewhat limited scenarios they support (basic image manipulation and hardware accelerated graphics).
At the end of the day, what we ship in box will never include stuff to satisfy everybody and dev's will need to create or depend on additional third party dependencies in some scenarios. There are benefits and drawbacks to this, just like there are benefits and drawbacks to tying your audio library and video library together, etc. Having smaller more actionable proposals to extend the existing stuff we expose is likely more actionable and while we can have larger discussions about more complete replacements, that is also a much bigger ask and has to be weighed against existing libraries that are likely doing it better (and likely will always do it better).
I 100% agree but look @ JCode's HomePage Github
That is why I started my library to enhance what issues they had at the time. It looks like they have also moved onto hardware acceleration where supported, which unfortunately I didn't get all the way to; I did get through Manged Intrinsic's though (just barely)
But I do again agree 100% (DRY and DNRTW)
That typed, it's en entire foundation outside of java... We probably need a .net Multimedia Foundation not to mention call indirect or abstraction over the PCI Bus using Pipes...
We are not that far away with Socket.IOControl honestly.... We could File.IOControl ( think there is already a proposal ) ...
We need that way before we have Color otherwise it does no good (unless your on Temple or Zenith OS)
aaronfranke
commented
4 years ago @tannergooding System.Drawing.Color uses 8 bits per channel, not 8 bits per pixel, and System.Numerics.Vector4 would give 32 bits per channel, not 32 bits per pixel.
tannergooding
commented
4 years ago Sorry, simple typo was thinking one thing and wrote another
juliusfriedman
commented
4 years ago It also limits you to 4 channels, in the purest form without another class to break out the components. my implementation makes no such limitation right now (only up to 255 channels because of byte but easily changed to int though).
There needs to be a structure of something right in front of the Vector and Behind the Color hence why I suggest Components and Buffers and a BitReading API after you have that it's just how many ops you need to do and in what order.
Otherwise we are just back as Unsafe.As and may as well just use Span to represent everything π¦
dotMorten
commented
3 years ago Since Color8 will be by far the most used color type, and HDR secondary (based on that the old 8bit color format has been sufficient for most things), I'd prefer this one is just called Color and the 16bit one would be renamed Color16 instead for the more special-case scenario.
I'm also a little confused why you're adding a second 4x8 color type. Why not just add the extra methods to System.Drawing.Color, and create a new System.Drawing.Color16 ?
aaronfranke
commented
3 years ago @dotMorten The proposed HDR color type is made of four floats, which ideally would be 32-bit single-precision floats (for 23 bits of mantissa) so that it can handle 16-bit HDR and more (23 >= 16), not 16-bit half-precision floats (which would give 10 bits of mantissa, and would not always work correctly for 12-bit and 16-bit HDR). Additionally, 16-bit integers would give high color depth, but would not be suitable for all types of HDR because it can't handle values above 100% brightness.
The idea of calling the float version Color and the byte version Color8 is that the former is more flexible, so it would make sense to be the "default" choice. However, the other way could make sense, but not as Color16, it would be Color32 (or maybe it would be better to call it ColorF, or something else).
dotMorten
commented
3 years ago Sorry yes Color32. ColorF also works and there's some presedence for that.
it would make sense to be the "default"
I'd say the other one should be default. Working with pixels quickly adds up to a lot of memory. In most cases quadrupling your memory footprint without realizing it when you most likely won't need HDR seems misleading. I'd rather you force the user to think about that by opting in with the non-"default".
charlesroddie
commented
3 years ago @dotMorten System.Drawing.Color has a really nasty memory inefficient implementation as noted in https://github.com/dotnet/runtime/issues/48615 . So another one is needed.
If this issue can be acted on before dotnet6 then it will save a lot of duplication because Maui.Graphics and Maui can then depend on it (link).
I agree with @dotMorten that 32bpc is excessive for a non-specialist Color-type. The consensus of this thread is that specialist color types belong in other libraries. Maui.Graphics currently uses a 32bpc Color type but that should be at most 16bpc.
aaronfranke
commented
3 years ago The consensus of this thread is that specialist color types belong in other libraries.
I disagree with that conclusion. A "specialist" 32bpc Color type has a very wide range of use cases, so I disagree that it's actually specialist, and I think it's worth including inside of .NET alongside an 8bpc type. Otherwise I agree, System.Drawing.Color should be replaced, and if we want to call the 8bpc one "Color" then that's fine with me as long as we have a 32bpc one available.
charlesroddie
commented
3 years ago A "specialist" 32bpc Color type has a very wide range of use cases
Can you give them? Only HDR has come up. And HDR is not only a niche case, but a niche within photography, and it doesn't make sense to support photography usage without a lot of extras - color profiles at least.
thargol1
commented
3 years ago Just my 2 cents:
Cons:
Pros:
Compromise I come from a county with 17 elected political parties, so compromise is in our blood. Maybe this can be implemented as an one or more extension packages? Just like sql-server support for EF-core is in a separate Nuget package.
Gavin-Williams
commented
2 years ago .. why the System.Numerics namespace? Not that I have a better suggestion
How about System.Graphics.
"DirectX and System.Drawing use ARGB
DirectX does not explicitly use ARGB, it uses all types. I use R8G8B8A8_UNorm (RGBA) typically for swapchain.
The major graphics players in the .NET ecosystem all have their own types already and this will not provide unification among them
If they choose to not adopt a communicable type, that's up to them, meanwhile, the up and coming graphics libraries - which we are in dire need of, could use this type. And anyway, system.numerics was generally adopted.
Just for clarification, however, would you feel the same way about having some "color" functions that operate on Vector4 exposed under the System.Numerics namespace (rather than exposing new types)?
Isn't this just an admission that we need a global color type, but instead of having a color type, telling people to use Vector4?
Vector4 also comes with the benefit of being recognized as a JIT intrinsic so it can be passed in a SIMD register automagically and has lots of useful math already defined.
I think one of the points of Microsoft implementing a color type would be to provide SIMD where appropriate to a natural representation. Which would be preferred over the current situation where people are implementing their own color types (myself included) that certainly don't have any optimizations.
Wider color gamuts and higher dynamic range will become the norm over the next few years, so by the time those additions would be useable by any number of developers, they'd be on their way to obsolescence.
I've just upgraded to a 10 bit / HDR600 display myself (up from 8 bit / HDR400) And yeah, from now on, my own efforts will equally consider that richer scenario. But don't RGBA(4byte) and RGBA(4float) handle all cases anyway for regular color handling - just putting aside the legacy formats BGRA / ARGB for the moment and the many variants of color information that can be useful in DirectX - the 4 float RGBA can handle any HDR or X-bit color precision (10 -12 bit) and the 4 byte RGBA is still useful for 8bit textures and vertex colors which don't need to be replaced, as 8bit textures can still be processed by 10bit / HDR pipelines, can't they?
Any color/pixel struct would implement interfaces that provide information about the pixel format, memory layout, color space, etc.
How does that work? I mean, I'm just not experienced enough to understand how you will use interfaces to describe formatting and color info. I would imagine if the struct is primitive and can be interpreted in different ways, then there might need to be a color-descriptor associated with any representation. Example,
class Texture<Color32>
ColorDescriptor descriptor; describes color space etc
Color32[ , ] data;You just can't include all the information in the color struct, and you don't want to.
The idea of calling the float version Color and the byte version Color8 is that the former is more flexible, so it would make sense to be the "default" choice.
Flexible maybe, but I wouldn't call it common or default. The common case is 8 bit textures, that's where people will start - loading and working with bitmaps and even when you get to feeding shaders, you're still using 8bit textures. Vertices will typically be defined with 8 bit color as well. Now that I actually have a 10 bit display I might start trying out something higher precision, or maybe I won't, maybe I'll just keep using 8 bit textures and leave it to the shaders to bring out the higher range of colors.
As discussed in https://github.com/dotnet/runtime/issues/14825, the existing
System.Drawing.Colortype is not ideal for all use cases and .NET Core could benefit from new color types.This proposal includes two new types:
ColorandColor8, in theSystem.Numericsnamespace in a new folder (library? package?) calledSystem.Numerics.Colors.Rationale and usage
The new
Color8type uses fourbytevalues. This is intended to be the computationally-cheap color type. It is fairly similar toSystem.Drawing.Color. TheColor8type is not suitable to use for HDR color math, including on high-end displays or in VR, but theColor8type is ideal to use on computers which display 8 bits per channel, without having any unnecessary bits. A color type using bytes can also be accelerated on GPUs when used on vertices.The new
Colortype uses fourfloatvalues. This is intended to be the fully capable color type, even if it's slower thanColor8and takes up four times the space in memory. The reason this is calledColorwith no suffix is because it should be presented to users as the default choice, since it is the most capable. This type can be used for HDR color math, since it supports values outside of the normal [0, 1] range, and it supports much higher color precision thanColor8. HDR implementations typically need 10, 12, or 16 bits per channel, floats have 23 bits of mantissa which is plenty. A color type using floats is also ideal when exposed to shaders (GLSL or similar).Both of these types have properties for setting with the other's member type (
R8/G8/B8/A8/Rf/Gf/Bf/Af), as well as operators and constructors to convert between them, so it is easy to interchange these types, including the ability to do high precision color math and store it in a low precision (byte) context, or vice versa.Additionally, this proposal includes
H/S/Vproperties for getting and setting hue, saturation, and value. This is a very common workflow for colors, especially when getting values from user input.Finally, there are
ColorsandColors8static classes which have preset colors. The existingSystem.Drawing.Colorplaces these as static members of that struct, but it is quite useful to have these separate to distinguish them from other static members (such asFromHSVand any others we wish to have).The uses of these types are too broad to include specific code examples. See the above text for information on how these types are expected to be used.
Proposed API
None of the properties listed below are auto-properties, all non-static properties read/write from the fields.
The proposed code for this API is collapsed, just expand it to view the code.
Details
While making this proposal, I referenced several open source color types:
Of course,
System.Drawing.Color(see docs). Some things fromSystem.Drawing.Colorwere not included or implemented differently. For example, instead ofGetHueetc methods,Hetc properties were used.Godot's
Colortype (see non-C# docs), MIT license. It usesfloatRGBA, and is very similar to the proposedColortype. Godot's strategy is to have one universal Color type that's used everywhere.Stride's color types (formerly Xenko), MIT license. Stride's strategy is to have tons of types and write tons of code for every use case, and there's a point of bloat and diminishing returns here. Stride has
Coloras using fourbytevalues andColor4as using fourfloatvalues, among several other types, but this isn't ideal and they have expressed that they're open to changing this.The proposed
Colortype is also very similar to MAUI'sColortype, which uses floats, has methods to convert between byte and HSV representations, and has a static classColorswith preset colors. It's also similar to Unity'sColortype, though this one is closed-source.When it comes to internal memory representation, string parsing, and importing from
uint/ulong, these types use RGBA order. Traditionally, Microsoft APIs such as DirectX andSystem.Drawinguse ARGB, but the rest of the world uses RGBA. Using RGBA order everywhere helps fit with the goal if "Semantic parity with CSS".Open Questions
Are there any common use cases that these color types are unsuitable for?
Are there any other methods or properties which should be added?
What are some good test cases to add?
Pull Request
None so far, but see https://github.com/dotnet/corefx/pull/40733 for a closed proposed (and outdated) implementation.
Updates
None so far.