Add differentiation functions

[AhmedMostafa16]

Background and Motivation

In mathematics, a derivative of a function of a real variable is another function that computes the sensitivity to changes in the output of the original function with respect to changes in the original function's arguments. Differentiation is the process of computing derivatives. Derivatives are a fundamental tool in calculus and have applications in many domains, including deep learning, robotics, Rendering, and ray tracing.

I can work on it with my pleasure.

Proposed API

public static double Derive(double f) {}

Usage Examples

int func(int x) 
{
    return x * x;
}
int result = Derive(func(3));
Console.WriteLine(result);    // Outputs: 6

Alternative Designs

N/A

Risks

N/A

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @tannergooding, @pgovind See info in area-owners.md if you want to be subscribed.

Issue Details

---

## Background and Motivation In mathematics, a derivative of a function of a real variable is another function that computes the sensitivity to changes in the output of the original function with respect to changes in the original function's arguments. Differentiation is the process of computing derivatives. Derivatives are a fundamental tool in calculus and have applications in many domains, including deep learning, robotics, Rendering, and ray tracing. I can work on it with my pleasure. ## Proposed API ``` C# public static double Derive(double f) {} ``` ## Usage Examples ``` C# int func(int x) { return x * x; } int result = Derive(func(3)); Console.WriteLine(result); // Outputs: 6 ``` ## Alternative Designs N/A ## Risks N/A

Author:	AhmedMostafa16
Assignees:	-
Labels:	`api-suggestion`, `area-System.Numerics`, `untriaged`
Milestone:	-

[danmoseley]

Thanks for the suggestion. What does the API look like on other platforms/libraries? Also, are there other related API that are required for it to be fully useful in .NET?

[AhmedMostafa16]

Hello @danmoseley. I wanted to add almost all linear differential equations to .NET. They would be implemented from scratch in .NET.

[danmoseley]

I guess what I'm asking is: if we add this API alone first, would it enable new scenarios or would it need more API? Incidentally just curious what algorithm would you expect to use to implement it?

[AhmedMostafa16]

Of course, it will enable new scenarios and expand the use of C# like:

• Animations: functions specify the rate of change of parameters for animations. Differentiation enables easy manipulation of these functions.
• Games: equations can be modeled using differentiable functions in game engines. Intelligent agents in games can be trained using techniques like machine learning that are enabled by differentiation.
• Machine Learning: Today, machine learning is predominantly done in dynamically-typed languages like Python, and Javascript: these languages are concise and easy to use. However, some people prefer safer programming: features like type checking and static diagnostics help catch errors early and improve productivity.
• Simulations: simulation techniques for fluids and other physical processes are based on approximate solutions to equations defined in terms of derivatives, like the Euler equations and Navier-Stokes. Being able to differentiate functions is an important building block for implementing algorithms to solve these equations.
• Robotics: algorithms used in robotics and mechanical engineering rely on (often higher-order) derivatives of functions that model the behavior of joints and other physical systems. A language like C# that can efficiently compute these derivatives without incurring the unpredictable runtime overhead of garbage collection may be well-placed to run aboard robots.
• Rendering and ray tracing: rendering systems are black boxes that consume data structures with scene geometry and produce images, but the physical processes they simulate are made up of differentiable functions. Building a ray tracer out of differentiable building blocks unlocks applications like inverse rendering (going from an image to scene geometry).

For what algorithm would I expect to use to implement it, I am really not sure because I want to check the system from all sides for no bugs and the highest possible performance.

[huoyaoyuan]

First, derivative is a operator of function. If it could be done, the api and usage should be:

Func<double, double> Derive(Func<double, double>);
double DeriveAt(Func<double, double>, double);

Derive(func)(3)
DeriveAt(func, 3)

But it's not possible for arbitrary function. We must get the expression of the function, then the api becomes:

Expression<Func<double, double>> Derive(Expression<Func<double, double>>);
double DeriveAt(Expression<Func<double, double>>, double);

[AhmedMostafa16]

Hello @huoyaoyuan, thanks for your API design. Your proposed API makes sense. In my snippet above I only added a tiny abstract snippet about the simplest type of differentiation, linear differentiation, but this means that this idea is good, isn't that? We can extend with attributes like [Derivable] attribute and adding other types of derivation.

[gfoidl]

linear differentiation

Is there a "non-linear differentiation"? BTW: you mean "derivative", and that's always linear as it the tangent to the curve at the desired point and is defined as dy / dx.

I believe a API like this shouldn't be added, as there are too many special cases to cover and a generalization is almost impossible. Just think about

• how to represent the function to be evaluated -- see @huoyaoyuan comment
• computation would be done by difference quotient -- how to chose the "step size" (i.e. divisor) and should it be just backwards or around the center / symmetric, in general: how to deal with condition
• what if the function isn't continuous or it's piece-wise defined
• and so on

Then where will this start and end. Do we need higher order derivates too or can / should they be expressed with these APIs? Multi-dimensional derivates would be nice too (and are very common for the listed use-cases). What about the inverse operation (integration)?

In essence I see this as specialized application that should be implemented occording the given domain / problem and not being generalized.

Perf-wise a very generic and general implementation would be bad too in contrast to an inline implementation of the difference quotient. So I doubt if this API is within .NET that it will be actually used in numerical software.

[AhmedMostafa16]

Hello @gfoidl, this snippet of code will explain a lot:

const double DX = 0.0000001;
double Derive(double x) 
{
    return Math.Round(((x + DX) - x)/DX);
}
double cube(double a)
{
    return a*a*a;
}
/*
cube(3.0) returns 9.0
Derive(cube(3.0)); returns 18.0
*/

Edit:

Differentiation is used in many different fields so why can't it be generalized?

[gfoidl]

Differentiation is used in many different fields so why can't it be generalized?

This question already contains the answer 😉. Because of "many differnt fields" that are specialized. And in reality it's not just x*x*x, but way more complex terms, then the trouble begins. Just think about another very simple function y(x) := ln(x).

For your example: It may be smarter to just use the derivate function direclty: 3 * x * x. Then -- as given by your snippet -- this results in 1. Look at your edit (the formula), so you see that the function needs to be passed and evaluated at x+dx.

Why is 0.0000001 the correct value for DX? How does this behave for rounding errors / condition to the problem? So how would one chose the defaults? Or when they should be passed in parameters they need to be validated (NaN, etc.) and this incurs a cost for each operation, thus being expensive.

BTW: more problems start by modeling the reality into a mathematical model with partial differential equations, which need to be solved. So the derivate is present in the basic-formulaes but not that often used in actually solving a problem (at least in the domain of engineering in which I reside).

---

cube(3.0) returns 9.0, Derive(cube(3.0)); returns 18.0

Please double-check these values.

[FiniteReality]

const double DX = 0.0000001;
double Derive(double x) 
{
  return Math.Round(((x + DX) - x)/DX);
}
double cube(double a)
{
  return a*a*a;
}
/*
cube(3.0) returns 9.0
Derive(cube(3.0)); returns 18.0
*/

I'm pretty sure your code and your image are showing two different functions.

Your code is currently showing this:

To make it show this:

You need to take f as a parameter:

private static readonly double h = 1e-6;

double ApproximateDerivative(Func<double,double> f, double x)
    => (f(x + h) - f(x)) / h;

That would produce a value which you expect (using f(x) = x ^ x as an example here):

[FiniteReality]

Personally, I think derivatives are more suited to a more maths-oriented framework which deals specifically with numerics, rather than the standard library, because there are many ways to algorithmically calculate the derivative of a function, and a simple Derive(x => x * 2, x) function is ignoring everything that goes into calculating that derivative.

[AmrAlSayed0]

I'm surprised Numerics.NET hasn't been mentioned. It has been around for a long time and is very mature.

[AhmedMostafa16]

Hello @AmrAlSayed0, I only wanted to add differentiation and some calculus features into the language features. So, I agree with your that the Numerics area is mature

[danmoseley]

https://github.com/Partydonk/partydonk/issues/2