Implements the PID+EMA algorithm as a torch.optim.Optimizer. The current implementation supports updates with dense and sparse gradients. The latter is required in the context of IndexedMultipliers with GPU execution.
Testing
Manual tests to ensure the correct updates were applied in a simple quadratic minimization problem for dense and sparse gradients.
Convergence test on a simple 2d constrained problem when using PID as the dual optimizer.
Co-authored by @juan43ramirez and @gallego-posada.
Changes
Implements the PID+EMA algorithm as a
torch.optim.Optimizer. The current implementation supports updates with dense and sparse gradients. The latter is required in the context ofIndexedMultiplierswith GPU execution.Testing
Co-authored by @juan43ramirez and @gallego-posada.