tl2622
commented
3 months ago Theory: I have mainly studied section 3.3 of the rough path lecture notes. Here is a summary. Main goal: optimize the loss function to achieve best fit method: compute the gradient of the loss function with respect to the parameters and then use SGD or adam to locate min
Neural ODEs and CDEs:
- direct back propagation through the internal operations of a differential equation solver (proof explained in other sections)
This method is known as discretise-then-optimise. Generally speaking, this approach is fast to evaluate and produces accurate gradients, but it is memory- inefficient, as every internal operation of the solver must be recorded.
- adjoint equation with two additional calls on ODE solvers (as two proofs are similar, proof will be explained only in the CDE section)
Adjoint-based methods are usually slightly slower to evaluate because one needs to recalculate yt in the backward pass. Also, they might introduce small errors compared to direct backpropagation through the ODE solver, so they should generally be used when memory is a concern.
I got roughly three applications of machine learning: 1. Hand-written number identification: this is basically a neural network function such that there are n*n pixels input and 10 numbers(0-9) outputs, we need to train this model by the concept called back-propagation to correct the parameters, finally in order to increase the precision of identification. 2. continuous dynamics of financial time series data, such as stock prices and market indices. Neural ODEs naturally capture the continuous evolution of market data by embedding neural network architectures within the differential equation solving process. This allows for real-time updates and predictions based on the latest market information, offering investors more accurate risk assessments and decision-making tools. 3. autonomous driving technology. Machine Learning combined with Ordinary Differential Equations (ODEs) is utilized to model and control vehicle dynamics accurately. ODEs help simulate the vehicle's motion through Newtonian mechanics, predicting future states based on current velocities and accelerations.