Create core geometry, drawing and imaging libraries

[govert]

This issue follows from the discussion at https://github.com/dotnet/corefx/issues/1563.

It is proposed to create a legacy-free library or set of libraries that would eventually be incorporated into corefx. The libraries would define primitive types and interfaces in the 2D and/or 3D geometry, drawing and image processing areas that are complementary to corefx. Such definitions would encourage compatibility of use and implementation across platforms and between libraries that target corefx in these areas.

The coreclr and corefx newly define SIMD-optimized types in System.Numerics that have special processing by the JIT compiler. These types include Vector2/3/4 and related Matrix and geometry types. The new primitive types defined for geometry, drawing and image processing could make use of these optimized types where appropriate.

Some of the use cases that would fall under this topic are:

• Image processing libraries for asp.net and other server scenarios
• Cross-platform 2D drawing libraries similar in scope to the Win2D library for Universal Windows Apps
• 2D and 3D geometry libraries that perform computational geometry or mesh processing

The proposal is not to provide comprehensive cross-platform implementations to cover these areas. Rather, the intention is to define common types and API patterns (perhaps in the style of OWIN for http handlers) that various implementations could align with. These libraries should consolidate work done for corefx that fall within the geometry, drawing and image processing scope.

There are many questions:

1. What are the use cases for such libraries?
2. What is an appropriate breakdown of the areas covered? How do the bitmap and vector worlds interact?
3. How wide should the scope be? Should it include color, fonts, text layout, UI? Now or later?
4. How should the new libraries relate to existing open-source implementations (single platform and cross-platform) in these areas (both pure .NET and managed wrapper libraries)?
5. What are the first steps and how should the effort proceed and be managed?

Can we make a list of existing .NET (open-source or otherwise) libraries in this space:

1. How popular and active is the library?
2. Is it likely to target corefx in the future?
3. What primitive types and interfaces would be relevant to that library?

[colombod]

Hi, this is a proposal i was sketching,

Api structure proposal

The goal si to provide an API structure to be asy to extend and implement for different domains. Applciation domains can be

• Graphics manipulation and rendering
• Algebraic algotiryrhms
• Statistic applications

Define vectors and matrices as pure algebraic objects

Operators should be focusing on the core requirements to be able to perform all basic operations. Being values those should be immutable, to avoid confusion when adopted in code. Operations should be clear and easy to extend. Different representation should achievable via casting.

If we consider some of the classic methods for intersection, dinstance and projection they can be used efficiently by changing conventions.

A rectangle can have different definition and most approaches distinguish between a generic box or axis aligned box pickinghte right one for the current application.

In a lot of application code is spent into translating the off the shelf rectangle to match the current need.

In this scenario going from cartesian point to polar point wouldbe performed via a cast operator obtaining a brand new point, operator will be specific to the type and some function can actually perform implicit transformations.

At this level is important to realise how different are the mathematical and geometrical domain. Is alwasy possible to add to float4 object but the sum of two points is not really defined (center of mass is actually a weighted average) while subtracting a point from another originates a vector.

Both points and vector can be constructed on a float array with arbitraty size (this could actually take advantage of the simd implementation already present in CoreFX).

[colombod]

Once the core math support is in place we could follow that to introduce colour definitions and manipulations, geometric primitives and operations in both 2D and 3D space. Rendering engines (plus image I/O) should be (in my opinion) the last mile covered. That should make possible to have windows based rendering and OpenGL one for cross platform development like iOS and Android. I would really appreciate help and input to shape such layers, I have tons of good intents but is really time the resource I am not plenty of.

[grokys]

Maybe listing projects that would be in scope to create replacement primitives for would be useful:

• WPF
• System.Drawing
• SharpDX (https://github.com/sharpdx/SharpDX)
• NGraphics (https://github.com/praeclarum/NGraphics)
• Perspex (https://github.com/grokys/Perspex)

Anything else?

[govert]

• Win2D (https://github.com/Microsoft/Win2D) is important, since it is the only existing library that will actually be targeting coreclr (if I understand correctly).
• In the image processing space, ImageProcessor (http://imageprocessor.org/) was an open-source server-side library mentioned before.
• Aforge.NET (http://www.aforgenet.com/) has a bunch of image processing stuff too.
• MonoGame (https://github.com/mono/MonoGame) does cross-platform 2D and 3D, so might conceivably want to target coreclr/corefx.

[dsplaisted]

• OxyPlot

[govert]

@grokys How would you imagine a set of primitives in corefx would be used in Perspex?

Suppose my Perspex app wants to use ImageProcessor to do some bitmap manipulation. Is it really important (or likely) that the Bitmap type would be unified? What are the desirable and plausible scenarios for you?

[colombod]

@govert I suppose that bitmap should be built on top of some more mathematical structure, part of CoreFx, all those libraries mentioned redefines the same types, the question is what would have been the common layer in CoreFx that could have made all of them interoperable?

[govert]

To partially answer my own question (and add another entry to our list), I found:

• Splat (https://github.com/paulcbetts/splat) - A library to make things cross-platform that should be.

This is what Perspex seems to use for a cross-platform Bitmap type. It has some orthogonal stuff, but for the purpose of this thread, it has Xamarin, WPF, WinForms WinRT packages that do:

• Cross-platform image loading/saving
• A port of System.Drawing.Color for portable libraries
• Cross-platform geometry primitives (PointF, SizeF, RectangleF), as well as a bunch of additional extension methods to make using them easier.

[grokys]

@govert - I've not considered usage in Perspex of anything more than the primitives (Point, Rect, Matrix etc), however in the long term I'd like to hand off the actual rendering etc to another backend if one becomes available (I've had my eye on NGraphics but my questions haven't yet been replied to).

If we could just get unified primitives that could be shared with e.g. SharpDX (how likely this is I don't know) it would be useful.

I don't use Splat for Bitmap, at the moment it's using its own implementation backed with the implementation from the rendering backend (e.g. Direct2D, Cairo). AFAIK Splat just tries to make the System.Drawing interfaces x-plat.

[grokys]

Maybe https://github.com/xoofx would be interested in this conversation (can you ping people from outside in an issue like that? Who knows).

[dsplaisted]

I'm pretty sure you can just ping them like this:

Maybe @xoofx would be interested in this conversation

[govert]

• Pinta (an open-source GTK# fork of Paint.NET ) - is a cross-platform .NET image editing program.

I would image that GTK# would be one of the back-end implementations that needs to be taken into account when deciding on primitives for corefx.

[KrzysztofCwalina]

cc: @Priya91

[JimBobSquarePants]

What kind of timescale are we looking at with creating this? There appears on the surface that there has been very little activity and progress in the repo over the last few months and I'm concerned that the library is not receiving the priority attention it requires.

I've been working on a cross platform c# version of ImageProcessor over the last few months and I seem to have made a lot more progress than here.

However, I simply don't know whether I should continue with my work or abandon it and wait for Microsoft.

It's all very frustrating.

[KrzysztofCwalina]

@JimBobSquarePants, I just saw your comments. We have nobody working actively on this. It's a interesting problem space, but it does not align with our current priorities.

[KrzysztofCwalina]

I will close this for now. Please let me know if anybody wants to work on it and I will reopen.

[JimBobSquarePants]

Thanks for the update @KrzysztofCwalina I'll keep doing what I'm doing then.