Deep learning and the frameworks supporting it rely on differentiation to train neural networks. The differentials are used during the back-propagation stage of the training to tune the parameters of the network.
Automatic differentiation is a technique that can produce the differential of a function. Tools that can process C code an produce differentials have existed for years, but over time, people have discovered that they can integrate this capability into their programming language.
I can not do justice to the field of automatic differentiation, how it works, why it works and the applications, so instead I defer on the capabilities and benefits to two documents:
DiffSharp Introduction - a library that we have been developing with the help of Güneş Baydin, one of the experts in the field. DiffSharp served as the inspiration for Swift’s own automatic differentiation capabilities.
Automatic differentiation is in particular useful to produce the proper graph code that can be consumed by frameworks like TensorFlow and Microsoft’s OnnxRuntime.
We believe that we can bring the teachings of DiffSharp and Swift into C# and .NET. This would require:
For the compiler to produce a differentiation function from a base function that has been tagged for differentiation.
Convenience attributes to flag structures or fields as being differentiable.
Convenience attribute to flag methods and properties to be differentiable.
Support on core libraries to annotate them for differentiability (arrays, dictionaries).
While initially we thought that this should be done by modifying the C# language and runtime - based on discussions with Mads Torgensen, we will first explore building this on top of the upcoming Source Generators capability for C#. If that approach works, we could implement everything as a secondary pass without needing to alter the core libraries or the C# language.
Adoption into C# is a longer-term play, we will need to first explore and prototype with DiffSharp and Swift to get a good sense of what needs to go into the compiler, as altering C# is vastly more complicated than the work that is needed to explore the space with F# and DiffSharp.
Automatic Differentiation
Deep learning and the frameworks supporting it rely on differentiation to train neural networks. The differentials are used during the back-propagation stage of the training to tune the parameters of the network.
Automatic differentiation is a technique that can produce the differential of a function. Tools that can process C code an produce differentials have existed for years, but over time, people have discovered that they can integrate this capability into their programming language.
I can not do justice to the field of automatic differentiation, how it works, why it works and the applications, so instead I defer on the capabilities and benefits to two documents:
Automatic differentiation is in particular useful to produce the proper graph code that can be consumed by frameworks like TensorFlow and Microsoft’s OnnxRuntime.
We believe that we can bring the teachings of DiffSharp and Swift into C# and .NET. This would require:
While initially we thought that this should be done by modifying the C# language and runtime - based on discussions with Mads Torgensen, we will first explore building this on top of the upcoming Source Generators capability for C#. If that approach works, we could implement everything as a secondary pass without needing to alter the core libraries or the C# language.
Adoption into C# is a longer-term play, we will need to first explore and prototype with DiffSharp and Swift to get a good sense of what needs to go into the compiler, as altering C# is vastly more complicated than the work that is needed to explore the space with F# and DiffSharp.