search

PumasAI / SimpleChains.jl

Simple chains
MIT License
234 stars 13 forks source link

feature request: hessian #77

Open jbrea opened 2 years ago

jbrea commented 2 years ago

How much work would it be to add something like valhessian!(H, schain, x, p)? Where H is the Hessian of a simple chain with loss.

chriselrod commented 2 years ago

It's possible to implement hessians (I'd suggest a forward over reverse approach), but would of course take some work.

A PR implementing that would gladly be accepted and I could provide instructions.

Long term, the plan for this library is to rely on Enzyme for its derivatives, in which case this shouldn't be a problem (in theory).

jbrea commented 2 years ago

Cool, thanks!

Currently I would only need hessians for chains with TurboDense layers. But I looked a bit into the code of dense!, and got a bit lost :)

Long term, the plan for this library is to rely on Enzyme for its derivatives

I was trying to use SimpleChains with Enzyme, but crashed julia (1.7.2). Should it already work with a newer version of julia?

chriselrod commented 2 years ago

I was trying to use SimpleChains with Enzyme, but crashed julia (1.7.2). Should it already work with a newer version of julia?

It probably shouldn't crash, but it also probably won't perform well until it can switch to the rewritten LoopVectoriziation. The rewrite is ongoing, and still a ways away, but the major reason it will help here is that it'll be able to run its optimizations after Enzyme generates the AD code, instead of before as is currently the case.

RS-Coop commented 7 months ago

Hello,

I am interested in training small models constructed with SimpleChains using second-order non-convex optimization methods, hence my interest in this thread.

I have a decent understanding of how to implement fast (mixed mode) matrix-free Hessian-vector products using Enzyme, and I am able to do this for regular Julia functions.

It seems that support for such functionality is not currently available (?), but this thread hints to it eventually being so. @chriselrod Can you comment on this? Thanks for the time!

Cheers, Cooper