Proposal for more System.Numerics APIs

[Perksey]

Introduction

System.Numerics.Vectors is a brilliant set of APIs for computing numerical vectors and matrices, however it has been lacking for a while. This has lead a number of people to whip up their own maths libraries (such as OpenTK and SharpDX, just to name a few) to fill in the gaps that System.Numerics left open. While #24168's proposal is certainly a step in the right direction and fixes the library's most glaring issue - the lack of double precision and other types of vectors - through "generic vectors". This is what I will be building on today.

Each part of this proposal is individually adressable, however I figured it would be better to contain all of it under one roof due to how closely related they all are.

More matrices

Summary

Currently, System.Numerics only contains 2 matrix types: a matrix containing three two-dimensional vectors and a matrix containing four four-dimensional vectors. This presents problems when users want to use matrices with different dimensions.

Use cases

As with a lot of mathematical stuff, the most notable use case for matrices with more dimensions are graphical applications. For example, when developing GPU-accelerated 3D applications such as games and game engines the user (when creating data structures to be passed to the GPU and shaders running on the GPU) will have to pass a Vector4 even if the W components aren't needed or used in any meaningful way. Matrix4x3 solves this as it allows the user to still send 4 vectors to the GPU within a matrix without needing to specify the W component (as this will likely be done in the shader). The same goes for Matrix4x2 and 2D applications.

Proposal

I propose the the following types be added:

• Matrix2x2
• Matrix2x3
• Matrix2x4
• Matrix3x3
• Matrix3x4
• Matrix4x2
• Matrix4x3

Mirroring their Matrix3x2 and Matrix4x4 counterparts

Box Geometry

Summary

Currently, there are no types providing axis-aligned bounding box (AABB) types, other than the System.Drawing types which do not use JIT intrinsics and do not use the System.Numerics types, making them awkward to fit into programs that are built around this library. I propose we add the following types:

Use cases

AABBs are commonly found in applications that need to do collision detection between 2D objects or 3D objects. For example, games and physics engines alike use AABBs to create a box around objects and detect when two AABBs collide. AABB have also recently been used in a number of ray-tracing APIs and applications, allowing users to pass in "acceleration structures" using AABBs to decrease the complexity of ray-intersection calculations.

Box2\<T>

public struct Box2<T> : IEquatable<Box2<T>>
{
    public T Min0;
    public T Min1;
    public T Max0;
    public T Max1;
    public Vector2<T> Min { get; set; }
    public Vector2<T> Max { get; set; }
    public Box2(Vector2<T> min, Vector2<T> max);
    public Box2(T min0, T min1, T max0, T max1);
    public Vector2<T> Size { get; set; }
    public Vector2<T> Center { get; set; }
    public bool Contains(Vector2<T> point);
    public bool Contains(Vector2<T> point, bool boundaryInclusive);
    public bool Contains(Box2<T> other);
    public T DistanceToNearestEdge(Vector2<T> point);
    public void Translate(Vector2<T> distance);
    public static Box2<T> Translate(Box2<T> input, Vector2<T> distance);
    public void Scale(Vector2<T> scale, Vector2<T> anchor);
    public static Box2<T> Scale(Box2<T> input, Vector2<T> scale, Vector2<T> anchor);
    public void Inflate(Vector2<T> toPoint);
    public static Box2<T> Inflate(Box2<T> input, Vector2<T> toPoint);
    public static bool operator ==(Box2<T> left, Box2<T> right);
    public static bool operator !=(Box2<T> left, Box2<T> right);
    public bool Equals(Box2<T> other);
    public bool Equals(object obj);
}

Box3\<T>

public struct Box3<T> : IEquatable<Box3<T>>
{
    public T Min0;
    public T Min1;
    public T Min2;
    public T Max0;
    public T Max1;
    public T Min2;
    public Vector3<T> Min { get; set; }
    public Vector3<T> Max { get; set; }
    public Box3(Vector3<T> min, Vector3<T> max);
    public Box3(T min0, T min1, T min2, T max0, T max1, T max2);
    public Vector3<T> Size { get; set; }
    public Vector3<T> Center { get; set; }
    public bool Contains(Vector3<T> point);
    public bool Contains(Vector3<T> point, bool boundaryInclusive);
    public bool Contains(Box3<T> other);
    public T DistanceToNearestEdge(Vector3<T> point);
    public void Translate(Vector3<T> distance);
    public static Box3<T> Translate(Box3<T> input, Vector3<T> distance);
    public void Scale(Vector3<T> scale, Vector3<T> anchor);
    public static Box3<T> Scale(Box3<T> input, Vector3<T> scale, Vector3<T> anchor);
    public void Inflate(Vector3<T> toPoint);
    public static Box3<T> Inflate(Box3<T> input, Vector3<T> toPoint);
    public static bool operator ==(Box3<T> left, Box3<T> right);
    public static bool operator !=(Box3<T> left, Box3<T> right);
    public bool Equals(Box3<T> other);
    public bool Equals(object obj);
}

Bezier Geometry

Summary

Many competing maths libraries contain bezier curve types which come in handy for many graphical operations such as vector graphics rendering, for example. Currently, the System.Numerics doesn't contain any curve types. Here are some proposed curve APIs:

Use cases

Bezier curves are used in all sorts of places. It's used by renderers of vector graphics (such as SVG files) and font renderers alike to achieve smooth curves. They also see use within the computer-aided design (CAD) segment - the concept of a "bezier curve" was actually adopted after Bezier was calculating smooth curves for Renault's cars.

BezierCurveCubic\<T>

public struct BezierCurveCubic<T>
{
    public T S0;
    public T S1;
    public T E0;
    public T E1;
    public T C00;
    public T C01;
    public T C10;
    public T C11;
    public T ParallelDistance;
    public Vector2<T> StartAnchor { get; set; }
    public Vector2<T> EndAnchor { get; set; }
    public Vector2<T> FirstControlPoint { get; set; }
    public Vector2<T> SecondControlPoint { get; set; }
    public BezierCurveCubic(T s0, T s1, T e0, T e1, T c00, T c01, T c10, T c11, T parallel);
    public BezierCurveCubic(Vector2<T> start, Vector2<T> end, Vector2<T> firstControl, Vector2<T> secondControl, T parallel = default);
    public Vector2<T> CalculatePoint(T t); // 0 <= t <= 1
    public Vector2<T> CalculatePointOfDerivative(T t);
    public T CalculateLength(T precision);
}

BezierCurveQuadratic\<T>

public struct BezierCurveCubic<T>
{
    public T S0;
    public T S1;
    public T E0;
    public T E1;
    public T C00;
    public T C01;
    public T ParallelDistance;
    public Vector2<T> StartAnchor { get; set; }
    public Vector2<T> EndAnchor { get; set; }
    public Vector2<T> ControlPoint { get; set; }
    public BezierCurveCubic(T s0, T s1, T e0, T e1, T c0, T c1, T parallel);
    public BezierCurveCubic(Vector2<T> start, Vector2<T> end, Vector2<T> control, T parallel = default);
    public Vector2<T> CalculatePoint(T t); // 0 <= t <= 1
    public Vector2<T> CalculatePointOfDerivative(T t);
    public T CalculateLength(T precision);
}

Closing

I hope that the CoreFX team take this into consideration. I work with vectors and matrices a lot, and as maintainer of OpenTK I was responsible for maintaining a maths library which fill these gaps (and am on the verge of writing another one for Silk.NET too.) My dearest hope is that System.Numerics.Vectors becomes feature-rich enough that there's no need for competing libraries to be developed, so I hope that this is helpful in removing fragmentation in this space if taken on by the CoreFX team. Thanks for reading.

EDIT 19:45 UTC: Added use cases

[ghost]

Tagging subscribers to this area: @tannergooding Notify danmosemsft if you want to be subscribed.

[tannergooding]

Thanks for the suggestion @Perksey.

It would be helpful if you could aggregate some real-world use cases for the various types and prior art where applicable. This will help centralize the information if this does eventually goto API review.

DirectX Math and GLM provide a couple but not all of these types and the same can be seen if you look in other places like Direct2D, Unity, or other libraries.

[Perksey]

I've edited to add some use cases, thanks for the suggestion.

[micampbell]

I was interested in pitching a Matrix3x3 as well. Here is my reasoning. Matrix3x3 should be useful in other situations where solving a small system of equations like Ax = b. One such example - still in computational geometry (since many of System.Numerics seems to be focused on geometry) is finding the point that is common to 3 planes. Beyond this, there are countless uses for solving small systems of linear equations of order 2, 3, and 4. With the code already written to invert a 4x4 matrix, it seems only natural to extend this for 2 and 3. Additionally, there are use cases in graphics where a full homogeneous coordinate transformation would be important for 2D systems - as the Matrix3x2 is intended to address. Filling out the 3x3 with the remaining projective terms could be used in keystoning (https://en.wikipedia.org/wiki/Keystone_effect) like the map tilt in Google Earth.

I would like to see Matrix3x2 removed - along with the apologies for internal functions like invert and determinant which are not possible on non-square matrices. Instead both Matrix3x3 and Matrix4x4 would include an "IsProjectiveTransform" boolean. This could be used to indicate whether the last column is used or not. A Matrix3x2 with IsProjectiveTransform false, would work just like Matrix3x2. Similarly for Matrix4x4, which would function as a Matrix4x3. Finally these should be make into readonly structs. For reference, I am currently making these changes in TVGL (https://github.com/DesignEngrLab/TVGL/blob/CS8Vectors/TessellationAndVoxelizationGeometryLibrary/Numerics/Matrix4x4.cs), which is a library that we are using for CAD/CAM applications.

[tannergooding]

I would like to see Matrix3x2 removed - along with the apologies for internal functions like invert and determinant which are not possible on non-square matrices. Instead both Matrix3x3 and Matrix4x4 would include an "IsProjectiveTransform" boolean. This could be used to indicate whether the last column is used or not. A Matrix3x2 with IsProjectiveTransform false, would work just like Matrix3x2. Similarly for Matrix4x4, which would function as a Matrix4x3. Finally these should be make into readonly structs.

None of this is possible. They are all non-trivial breaking changes. It is also undesirable to use larger than necessary matrices in some scenarios and various libraries (such as DirectX Math and GLM) frequently provide types like Matrix3x2 and Matrix4x3 and even expose functions that may be useful but only sometimes possible, as they are intended for use in places where an approximation is "good enough".

[micampbell]

I suppose that makes sense. Still, a case for Matrix3x3 and Matrix2x2 could be made. I'm not prepared to properly do that at this time, and I don't want to distract from the plan of the OP, @Perksey .

[Perksey]

@micampbell Thanks for chipping in here.

All I’m after is System.Numerics.Vectors to become feasible for most scenarios out there, whereas today it’s currently lacking. As such it has (and continues to) lead developers to create their own spin-off libraries and that’s really what I’m trying to prevent - no one likes fragmentation on something as simple as this.

In any case, this will have to get past API review - something that has proven challenging (the original proposal for double precision vectors has been open for 3 years with continuous attention and no progression) because their priorities are elsewhere, so the ball is in their court and needs a nudge before we can pitch in. I would fully expect this to take a while though, and appreciate that Microsoft has to spend their time with other things instead of these proposals.

[prollin]

An argument can be made for more matrix types. The rest seem to be outside of what such a core API should provide; for example AABB can be represented in multiple ways: Min/Max or Center/Extent, the later being more efficient for intersection tests.

System.Numerics.Vectors is indeed lacking. I do think it should be limited to provide access to SIMD types and operators, anything more should be extension methods or part of a helper class (Math?).

As I mentioned in #24168, i think making those types actual primitive types would be ideal (and move all the math stuff in Math), though I have no idea how far fetch this is (or if it is even possible).

[mattleibow]

Hi folks! Love this issue! I need this for SkiaSharp https://github.com/mono/SkiaSharp/issues/1330

SkiaSharp is a 2D drawing engine, and we have been using our own types. And everyone wants to use Numerics. But we can't because we don't have all the types we need.

It would be great to be able to use BCL-based, intrinsics-powered matrices. Even if just the 3x3 and 4x4.

[Perksey]

Hi Matt, the .NET team have a lot on their hands at the moment and seem to be prioritizing the backlog, so don't be surprised if this takes another 3 years to get in!

Numerics-wise, we're currently busy implementing the new generic vectors which will support both float and double. The generic vectors are likely going to be in for .NET 5 in some form (whether hardware-accelerated or not) as there's a lot of backing by both the community as well as some .NET team members. Sadly I don't forsee any numerics proposals getting in for .NET 5.

I need to edit this issue a bit so it's constructed properly, hopefully then tanner will be able to mark it as api-ready-for-review and then we play the waiting game for it to finally make it to review.

[mattleibow]

I understand the work. I've been on some of the calls.

But this raises the question... I am wanting to use the types across the native boundary... What will the new generics mean for this? Will it be possible to just use as is if the APIs match? Can generics be used via p/invokes?

[Perksey]

Double vectors and float vectors with both be blittable, the generics don't change that as we throw an exception if you attempt to use or contruct any of the System.Numerics types using any T that isn't supported (currently only float and double), similar to how Vector\<T> works today.

However, you won't be able to take a pointer to Vector4\<T> and friends. You will need to know that the T is either float or double as the constraint on those types are struct rather than unmanaged, so you will only be able to take a pointer to a Vector4\<T> whose T is guaranteed to be unmanaged.

[tannergooding]

For the simple types like Vector2/3/4, Matrix3x2, and Matrix4x4, the types will be useable in interop code. They reasonably only have a single underlying representation. For types like Plane and Quaternion it's likely that the underlying implementation won't change, but they (particularly Plane) have multiple underlying representations across the industry, so it isn't strictly guaranteed that will always be the case.

Generics are allowed in interop code since .NET Core 3.0, you can likewise take the address of a generic provided it meets the unmanaged constraint. Given the constraint is where T : struct, you'll be able to have Vector2<float>* or Vector2<double*>, but not Vector2<T>*.

[jhm-ciberman]

However, you won't be able to take a pointer to Vector4 and friends. You will need to know that the T is either float or double as the constraint on those types are struct rather than unmanaged, so you will only be able to take a pointer to a Vector4 whose T is guaranteed to be unmanaged.

@Perksey Can you elaborate why do you want Vector4<T> to be where T : struct instead of where T : unmanaged? I don't see any benefit in allowing managed data inside a Vector4 that is meant only for graphics programming and numeric calculations.

by the way, I'm all in with this proposal 🚀 As a simply developer starting with graphics programming, not having a consistent api to work with AABB/Beziers/Matrices is a headache.

[richard-sim]

don't be surprised if this takes another 3 years to get in! ... I need to edit this issue a bit so it's constructed properly, hopefully then tanner will be able to mark it as api-ready-for-review and then we play the waiting game for it to finally make it to review.

So, we're now 3 years on, and I see some nice improvements being made to the Matrix3x2 and Matrix4x4 types for .net8.0 - yay!

Unfortunately, it looks like this proposal never made it to being api-ready-for-review, so I'm guessing there's no hope for even just the essential (from a 3D graphics perspective) Matrix3x3 and Matrix4x3 types to make it in with the other Numerics changes going into .net8.0. Another 3 years?