Plan for Deep Learning in .NET

[briacht]

Plan for Deep Learning in .NET

This past year we've been working on our plan for deep learning (DL) in .NET which is outlined in this issue below.

We came up with this plan by looking at recent deep learning trends, staying in line with current Microsoft strategy around deep learning, and gathering a variety of customer evidence (which you can read about in the Customer evidence section below).

Our goal is to execute this plan by the end of 2022.

Feedback

We would love to hear your feedback about our plan and whether this will fulfill your requirements for deep learning in .NET!

You can leave your feedback directly on this issue.

The plan

The main stages of the plan are:

1. Make it easier to consume ONNX models in ML.NET using the ONNX Runtime (RT)
2. Fully support and productionize TorchSharp for building neural networks in .NET
3. Build a bridge between TorchSharp and ML.NET

As part of this plan, ML.NET will continue to be more top-level and scenario focused, while TorchSharp will power the Deep Learning training and features in .NET.

As we iterate and work on the plan, we will make updates to this issue, including linking to relevant issues and PRs.

Stage 1: Make it easier to consume ONNX models in ML.NET with the ONNX RT

In ML.NET, you can already consume pre-trained ONNX models via the ONNX RT.

However, we have seen many issues from customers around the complexity of ONNX model consumption in ML.NET, particularly when trying to figure out the inputs and outputs of the ONNX model (which are required for consumption in ML.NET).

So, we plan to:

• Expose internal functionality to get the input and output schema from ONNX models (#5917).
• Add documentation outlining step-by-step how to get the inputs and outputs of any ONNX model and then how to consume that ONNX model in ML.NET. We will try and cover a variety of scenarios (#25863).
• Partner with the ONNX RT team to bring relevant ONNX RT advancements to ML.NET.

Stage 2: Fully support and productionize TorchSharp

TorchSharp is an API that provides .NET bindings for the PyTorch engine.

TorchSharp will be the API-based Neural Network training and inference in .NET; the custom deep learning training that you can do in PyTorch, you will be able to do in .NET with TorchSharp.

As mentioned in the TorchSharp repo, TorchSharp will retain Python naming conventions. This makes it very easy to port PyTorch to .NET TorchSharp as well as to bring Python examples to .NET.

In order to bring TorchSharp to production, we will:

• Move TorchSharp from the Xamarin repo to the .NET Foundation repo
• Add missing features and optimizers from PyTorch 1.6-1.9
• Update infrastructure for signed builds, tests, and SDLs

Ongoing support work will include:

• Issue and PR management of the repo
• Reacting to new versions of PyTorch (~2x a year)

Stage 3: Build a bridge between TorchSharp and ML.NET

Once TorchSharp is fully productionized, it will power higher-level ML.NET deep learning APIs which will follow .NET naming conventions and "feel" like .NET.

We will surface the following through ML.NET (powered by TorchSharp):

• Scenario-focused APIs for transfer learning, like the Image Classification API; there will be an API for each scenario, like Object Detection and NLP
• Generic transfer learning API for custom scenarios that don't fit the scenario-focused APIs
• Simplified ML.NET APIs and tooling for building neural networks from scratch (this will take a bit more investigation and planning)

Here is a diagram which visually outlines the transfer learning part of the plan:

Out of scope

For this plan, it is currently out of scope to port over the PyTorch domain libraries (Torch Vision, Torch Audio, and Torch Text).

Customer evidence

After talking to a variety of .NET customers, we discovered the following:

• Lack of deep learning support is a top pain point/blocker for using ML.NET
• Customers are forced to turn to Python-based frameworks for training DL models when scenario is not covered by ML.NET (mainly TensorFlow, Keras, and PyTorch)
• Customers have a variety of DL scenarios, the most popular of which include Image Classification, Named Entity Recognition (NER), NLP-based scenarios, and Object Detection
• Almost all customers said they prefer their E2E machine learning workflow, from training to consumption, to be in .NET, and they don’t care about the underlying framework ML.NET uses for training deep learning models
• A majority of customers we talked to want ability to build neural networks from scratch using .NET
• Several customers said adding transfer learning for their scenario would unblock them
• One customer we talked to needed better support for consuming pre-trained models which is also evident in the number of issues filed around consuming ONNX models.
• Customers were split in terms of tooling and automation: some prefer a code-first / API approach when building neural networks from scratch, while some prefer tooling.
• Customers were also split in terms of automation when training deep learning models: some prefer no automation and full control when training, some want full automation, and some want a combination of automation with a lot of control

Additionally, the following issues filed in the repo add further evidence for backing up our deep learning plan:

Image segmentation / regression

• #5276
• #5595
• #5595

NLP

• #5159
• #5874
• #955

Local object detection

• #5863
• #1091

Torch support

• #4053
• #3439

ONNX consumption

• #5810
• #5855
• #5830
• #5597
• #5484
• #5549
• #4578
• #4335

Other

• #5734
• #5713
• #5372

[justinormont]

Simplified ML.NET APIs and tooling for building neural networks from scratch (this will take a bit more investigation and planning)

You can take a look at how we exposed CNTK in TLC. Two styles, (1) fully manually designed NNs [do it myself], and (2) a simplification layer which created a network to your specifications [do it for me]. Both styles create custom NNs from scratch (no transfer learning).

For 2, you specify the network type you want created, for example a fully-connected (aka feed-forward) NN, and specify the parameters of the network, like number of hidden layers, number of neurons wide, learning rate, training algo, etc. Then we create the NN for you, including your specific input/output sizes (which is a bit difficult to know in TLC / ML․NET before training).

Exposing common neural network styles in a parameterized form makes it easy for a non-experienced user to create a functional NN, and also lets the AutoML do hyperparameter optimization easily on these networks (e.g. automatically alter the number of layers and learning rate in a fully-connected NN).

[sps014]

Great roadmap, i am excited for this.

[DFMERA]

Grate roadmap!! Simplifying the consumption of ONNX models is so necessary, I trained an ONNX model once but I could never understand the outpts. Thank you for adding this to your roadmap!!

[dsyme]

This is fantastic! Great to see the plan coming together :)

[pjmlp]

The roadmap looks great! Specially not having to deal with C++ frameworks directly and being able to live in pure .NET workflows.

[saint4eva]

The roadmap looks great! Specially not having to deal with C++ frameworks directly and being able to live in pure .NET workflows.

You are avoiding dealing with C++, but now you are going to deal with python. Performance is going to be lost and the API convention of python will overwhelm .NET and C# developers.

[dsyme]

...deal with python... performance is going to be lost

No Python is used in TorchSharp

[aforoughi1]

The roadmap should address the Model Explainability, built from bottom-up as a major milestone.

I found the AutoMl.Net framework significantly improved the rapid modelling process, so I hope to see it as part of the roadmap.

[pjmlp]

You are avoiding dealing with C++, but now you are going to deal with python

ONMX runtime is written in C++.

I am also quite comfortable with C++, including being up to date with C++20, but rather spend my time with languages where productivity comes first.

[nietras]

I'm happy to see authoring DL models in C# is finally being prioritized. I wanted to share another idea for this just for context.

Instead of using TorchSharp one could author models in ONNX format via OnnxSharp (https://nietras.com/2021/03/20/introducing-onnxsharp/) if that had an easy to use API for that. With a proper C# API, not the pythonesque TorchSharp API.

Training could then be done via the ONNX Runtime Training API. Or even TorchSharp if that could load ONNX models.

If I had time I would be working more on fulfilling this idea.

Having both options would be best of course. Important thing is if ML.NET settles on using ONNX format, there are a lot of options. So happy with that too.

[kinfey-zz]

Apple silicon support is important

[GeorgeS2019]

@nietras I am aware of and follow your contributions through OnnxSharp Keep it up!!!

Or even TorchSharp if that could load ONNX models.

TorshSharp save/load ONNX models has been discussed. We need more people to join in to discuss and support this idea.

[redradist]

@briacht

As mentioned in the TorchSharp repo, TorchSharp will retain Python naming conventions.

I like this one !!

I've asked the same in .NET MAUI:

https://github.com/dotnet/maui/issues/522

to retain as option the same native API naming conventions, but they did not listen to me ... (

[redradist]

@briacht

Why not to use also https://github.com/SciSharp/TensorFlow.NET ? It is actively developed and have the same "native" api as TensorFlow ...

Also there are Miguel package https://github.com/migueldeicaza/TensorFlowSharp ... but he does not evolve it unfortunately

[GeorgeS2019]

@redradist Tensorflow.NET already integrated in ML.NET TensorflowSharp is no longer as actively developed as Tensorflow.NET

[redradist]

@GeorgeS2019

@redradist Tensorflow.NET already integrated in ML.NET

What ?? When ? Where ?? I tried to find it in repo ... do not see it (:

[GeorgeS2019]

@redradist You can get ML.NET book e.g. from Amazon which assist in using Tensorflow.NET in ML.NET

[redradist]

@GeorgeS2019 Thanks, but as for me documentation is more important ;)

[redradist]

@GeorgeS2019 Seems like I've found it ;) https://github.com/dotnet/machinelearning/tree/main/src/Microsoft.ML.TensorFlow

[RoudyES]

This plan tackles almost every problem that most of us have when it comes to using ML.NET for DL, it is very exciting! Thank you for listening to your community needs, and, I really hope this plan comes to fruition without many obstacles.

[saint4eva]

...deal with python... performance is going to be lost

No Python is used in TorchSharp

Why python-like apis then?

[migueldeicaza]

...deal with python... performance is going to be lost

No Python is used in TorchSharp

Why python-like apis then?

Those are two separate issues.

TorchSharp is pure C# code calling into the native engine that powers PyTorch.

It also happens to have adopted the shape of APIs of Python, to simplify the job of moving code from Python to C#

[chunky]

Please can Reinforcement Learning be formally added as a goal under the banner of "Deep Learning Support"?

I put a ticket on modelbuilder about a year ago, here: https://github.com/dotnet/machinelearning-modelbuilder/issues/1026 The first ticket I can find in this repo is now 3 1/2 years old: https://github.com/dotnet/machinelearning/issues/181

[Jiuyong]

I'm happy to see authoring DL models in C# is finally being prioritized. I wanted to share another idea for this just for context.

Instead of using TorchSharp one could author models in ONNX format via OnnxSharp (https://nietras.com/2021/03/20/introducing-onnxsharp/) if that had an easy to use API for that. With a proper C# API, not the pythonesque TorchSharp API.

Training could then be done via the ONNX Runtime Training API. Or even TorchSharp if that could load ONNX models.

If I had time I would be working more on fulfilling this idea.

Having both options would be best of course. Important thing is if ML.NET settles on using ONNX format, there are a lot of options. So happy with that too.

I support your idea, but most people will feel that it's too radical. And cutting the ecology of torch is a double-edged sword.

[saint4eva]

I'm happy to see authoring DL models in C# is finally being prioritized. I wanted to share another idea for this just for context. Instead of using TorchSharp one could author models in ONNX format via OnnxSharp (https://nietras.com/2021/03/20/introducing-onnxsharp/) if that had an easy to use API for that. With a proper C# API, not the pythonesque TorchSharp API. Training could then be done via the ONNX Runtime Training API. Or even TorchSharp if that could load ONNX models. If I had time I would be working more on fulfilling this idea. Having both options would be best of course. Important thing is if ML.NET settles on using ONNX format, there are a lot of options. So happy with that too.

I support your idea, but most people will feel that it's too radical. And cutting the ecology of torch is a double-edged sword.

Cutting the C# /. NET ecosystem is even a more dangerous double-edged sword. Writing python and forcing C# developers to understand and use python.

They should take a look at what OpenCVSharp is doing. Beautiful apis that conforms to dotnet way of doing things

[lostmsu]

This is not doing reinforcement learning using ML.NET. The training happens in Python.

[chunky]

@JimFFM I already have a workflow that works well, that makes a round trip through Python [and has a more performant and less byzantine; but also less capable; ecosystem around it]. But I'd really like to skip said round trip and do it directly.

[torronen]

"1. Make it easier to consume ONNX models in ML.NET using the ONNX Runtime (RT)"

Could this fit the use cases for model builder?

1. Select ONNX model
2. Model Builder wraps everything in a zip file
3. Model Builder creates the consumption code and samples

I think it would make sense to help .NET devs consume ONNX models, even without knowing ML.NET libraries. I believe the data scientist creating the model may not be the one integrating it with the app. Model Builder might help overcome this gap between ML domain and application architecture domain. Maybe same feature could also support creating consumption samples for ML.NET .zip models. Data scientist could build the model in any chosen way and then provide the app dev with the models.

[WhitWaldo]

I'm happy to see authoring DL models in C# is finally being prioritized. I wanted to share another idea for this just for context. Instead of using TorchSharp one could author models in ONNX format via OnnxSharp (https://nietras.com/2021/03/20/introducing-onnxsharp/) if that had an easy to use API for that. With a proper C# API, not the pythonesque TorchSharp API. Training could then be done via the ONNX Runtime Training API. Or even TorchSharp if that could load ONNX models. If I had time I would be working more on fulfilling this idea. Having both options would be best of course. Important thing is if ML.NET settles on using ONNX format, there are a lot of options. So happy with that too.

I support your idea, but most people will feel that it's too radical. And cutting the ecology of torch is a double-edged sword.

Cutting the C# /. NET ecosystem is even a more dangerous double-edged sword. Writing python and forcing C# developers to understand and use python.

They should take a look at what OpenCVSharp is doing. Beautiful apis that conforms to dotnet way of doing things

I get where you're coming from, but I'd urge you to consider the simple reasoning for the design decision. Put simply, given the enormous amount of existing documentation, tutorial and guide content and other collateral already created for PyTorch, keeping the naming and API conventions identical makes all this applicable to the .NET bindings as well.

No doubt there'll be an opportunity down the road to make a .NET-first API on top of the work done here, but in the interest of not starting entirely from scratch, this approach makes sense to me.

[Regenhardt]

Honestly I think this might not be a great reason.

Assuming there will be many people actually porting python code over to ML.NET, after that is done everyone will have to continue using python naming conventions for no reason.

With new people coming to machine learning every day though, I think many people will be confused by the different naming, which makes it look like an interop wrapper around a python library, which it is not. It's its own thing.

I'm currently using python for my ML projects, but I've been trying to get into ML.NET hoping for some performance boost but also an easy way to build different frontends quickly, but most of all the static typing. Since we can build dotnet apps with almost no boilerplate and can even write dotnet interactive notebooks now, the IMO few advantages of python rapidly dwindle.

I'd immensely prefer the naming also being in dotnet convention, matching the rest of the framework.

[GeorgeS2019]

Out of scope

For this plan, it is currently out of scope to port over the PyTorch domain libraries (Torch Vision, Torch Audio, and Torch Text).

@briacht @luisquintanilla @migueldeicaza These out-of-scope topics defined back in Aug 30, 2021 are now being actively developed.

• Torch Vision
• Torch Audio
• Torch Text will be next :-)

Sept 2023 Update

.NET for pytorch TorchRL using TorchSharp [WIP]

[GeorgeS2019]

Now TorchSharp's TorchVision and TorchAudio have their own nugets

nuget Torchsharp TorchVision

nuget Torchsharp TorchAudio

[GeorgeS2019]

Sept 2023 Update for Deep Reinforcement Learning

@migueldeicaza