ChainRules support

[ChrisRackauckas]

The reason for the p in the optimization problem definition is to allow for ChainRules support. Specifically:

https://arxiv.org/abs/1804.05098 https://web.stanford.edu/~boyd/papers/pdf/diff_cvxpy.pdf https://link.springer.com/article/10.1007/s12532-012-0043-2 https://link.springer.com/article/10.1007/BF02055196 https://arxiv.org/abs/1810.13400

https://link.springer.com/article/10.1007/s12532-012-0043-2 equation (3) might be the clearest.

[gaurav-arya]

A note: special case differentiations for the Lasso / "proximal gradient methods" (e.g. https://arxiv.org/pdf/2201.12348.pdf, https://arxiv.org/pdf/2105.15183.pdf) might pop out automatically from chain rules here, after doing an epigraph formulation of the nondifferentiable parts of the objectives.

[ChrisRackauckas]

Unconstrained optimization is demonstrated here: https://github.com/gdalle/ImplicitDifferentiation.jl/blob/main/test/1_unconstrained_optimization.jl