Implement "blackbox" solvers as fallback if sympy analytic fails

lkeegan commented 5 years ago

would be useful for testing
calculate Jacobian numerically with difference approx
would also work with if statements, verbatim blocks, etc
performance not critical

pramodk commented 5 years ago

@lkeegan : could you add some reference framework / pseudo-code for implementing this one?

lkeegan commented 5 years ago

@pramodk yes will do

pramodk commented 3 years ago

@cattabiani : could you take a look at this ticket and understand overall scope off this ticket by reviewing existing solver implementation and references issues?

The overall idea is that if sympy fails then we need a fallback solver that we can rely on. This is needed to deprecate mod2c completely.

cattabiani commented 3 years ago

@pramodk do we already have a mod file where sympy fails?

pramodk commented 3 years ago

@cattabiani : off the top of my head I don't remember any mod file which was failing with sympy solver. But may be you can build artificial one ? You can also see some motivation in #55 and #85.

cattabiani commented 3 years ago

I am still reading literature about the latest methods around. Looks like numerical/automatic differentiation is used extensively in deep learning and there are already many tools available. Automatic differentiation looks superior (for a computer program) than the numerical one and avoids a lot of the pitfalls. Now the question is: what tool do we want to use?

A summary of what I have found:

autograd: still maintained but not developed anymore (good reviews)
tensorflow, pytorch quite big projects. Do we need cannons?
auto_diff very recent and probably developed by a very small team

if you have experience on the topic I am happy to hear your ideas @ferdonline @pramodk

pramodk commented 3 years ago

if you have experience on the topic

no...not much! I wonder if @ohm314 remember anything on this topic because IIRC he had discussions.

tensorflow, pytorch quite big projects. Do we need cannons?

certainly not!

@lkeegan : could I ask you if you remember what were were planning and possible ways to implement this? (of course if you remember)

cattabiani commented 3 years ago

After some research (thx @ohm314 ) here are a few notes.

Functions with an infinite ammount of points in which they are not C1 are probably of no interest for scientists
The "big" cases that are not completely handled by nmodl yet are:
- control flow blocks
- verbatim blocks inside derivative ones
- ?
Automatic Differentiation pares the evaluation of a function in a point with the evaluation of its derivatives (in that point). Basic operations are derived symbolically and computed in loco to simplify the expression. There is a very nice example in chapter 3 that explains my reasoning much better. This has the following advantages:
- precise up to machine precision (like symbolic differentiation)
- no "equation swelling" (better than symbolic differentiation and not prone to a possible "early sympy surrender" due to equations that are too long)
- no problem with control flows (and I guess with verbatim blocks) (numerical differentiation in these cases could be very difficult/impossible to implement correctly)
eigen contains routines for numerical and automatic differentiation: I am going to check how do they work (since we already ship eigen with coreneuron, this could be basically free for grab)

For these reasons, I am going to check how AD in eigen works exactly and, then, try to use it in case of control flow blocks and nonlinear system, which, by its structure, is already "in shape" to have eigen AD plugged in (I think)

lkeegan commented 3 years ago

@pramodk the idea was to have a version of the newton solver that used a simple numerical differentiation approximation to the jacobian as a fall back for the cases where sympy could not be used to provide the analytic jacobian (e.g. verbatim blocks), which was implemented here: https://github.com/BlueBrain/nmodl/blob/master/src/solver/newton/newton.hpp#L80-L152 I think if the sympy pass fails nmodl should use this version of the newton solver instead, although I can't remember if this was fully implemented - I remember getting lots of help from @ohm314 on this so maybe he remembers.. But if it would be possible to replace this with an AD method that would definitely be an improvement!

cattabiani commented 3 years ago

I checked and I did not find any practical case where sympy analytic can fail. AD is better in case of "equation swelling". However, for hand-written equations that problem is minor/non-existant. Thus, I close this issue. If there are any real case scenario appearing I will reinvestigate this issue

BlueBrain / nmodl

Implement "blackbox" solvers as fallback if sympy analytic fails #60