The `callback` appears to be called for linesearch iterations

[goerz]

The documentation of callback says

The callback function callback is a function which is called after every optimizer step

However, it appears to me that callback (at least for some optimizers like NLopt.LD_LBFGS()) is called for all evaluation of the loss function, which includes evaluations inside a linesearch.

See https://discourse.julialang.org/t/questionable-zygote-gradients-for-quantum-optimal-control-problem/112060/6?u=goerz, which I've since double checked to verify that indeed, the calls of the loss function and the calls to callback match up.

This behavior isn't really what I would understand as "called after every optimizer step" (since linesearch iterations are arguably internal to a single step of the optimizer). I mean, a callback for every evaluation of the loss function could be useful too in some circumstances, but it prevents things like checking for monotonic convergence. If you really wanted to go overboard, there could be an additional flag to choose what the callback behavior should be.

P.S.: I can provide a MWE, some variation of https://gist.github.com/goerz/e35a8418edd9d358f9dbaa6972c143b5#file-mwe_zygote-jl if that helps, but the issue is probably more on a conceptual level.

[ChrisRackauckas]

We should try and at least have a queryable way of knowing which behavior and optimizer is giving you, since it's algorithm-dependent.