zhenglei-gao / KineticEval

An R package for kinetic evaluations
4 stars 4 forks source link

Use `numDeriv` to calculate a more accurate hessian? #23

Closed zhenglei-gao closed 10 years ago

zhenglei-gao commented 10 years ago

It is possible that the hessian computation in optim algorithm used is not sufficiently accurate. We could add a step to calculate hessian at the optimal point.

Shall we???

zhenglei-gao commented 10 years ago

How I can classify stationary to extreme points or non-extreme points depending on hessian?

zhenglei-gao commented 10 years ago

After you get the hessian matrix, you have to calculate it's eigen values, if all of them are positive then you have a minimum, if all are negative, then it is a maximum, if some are negative and some are positive then you don't have any of the previous one's, in this case you might have a saddle point (imagine a horse saddle-like surface).

In the case that you at least one null eigen value then, to find out what kind of stationary point it is, you will most likely have to calculate the taylor expansion of the function and see how the function varies with that approximation.

zhenglei-gao commented 10 years ago

The Hessian being 0 and its determinant being 0 are also not the same thing.

Basically, if the Hessian determinant is 0, the higher-order terms may be needed to determine the classification of a critical point, but this is not at all trivial to do in general .

If the gradient (the vector of the partial derivatives) of a function f is zero at some point x, then f has a critical point (or stationary point) at x. The determinant of the Hessian at x is then called the discriminant. If this determinant is zero then x is called a degenerate critical point of f, or a non-Morse critical point of f. Otherwise it is non-degenerate, and called a Morse critical point of f. Hessian Matrix play an important role in Morse theory, because its kernel and its eigenvalues allow to classify the critical points.

zhenglei-gao commented 10 years ago

A bordered Hessian is used for the second-derivative test in certain constrained optimization problems

zhenglei-gao commented 10 years ago

Nearly Singular Hessian

The basic problem of a singular Hessian is almost always due to a non-identifiable parameter, so that the p-dimensional objective function really lives in (p-1) (or fewer) dimensions. could be linear or nonlinear dependencies, extremely sparse data causing the algorithm to fail, bad starting values, etc..

the second derivative test is inconclusive, and the point (a, b) could be any of a minimum, maximum or saddle point.

zhenglei-gao commented 10 years ago

the asymptotic theory behind the formula for 's.e.' breaks down with parameters at boundaries. It assumes that you are minimizing the negative log(likelihood) AND the optimum is in the interior of the region AND the log(likelihood) is sufficiently close to being parabolic that a reasonable approximation for the distribution of the maximum likelihood estimates (MLEs) has a density adequately approximated by a second-order Taylor series expansion about the MLEs. In this case, transforming the parameters will not solve the problem. If the maximum is at a boundary and if you send the boundary to Inf with a transformation, then a second-order Taylor series expansion of the log(likelihood) about the MLEs will be locally flat in some direction(s), so the hessian can not be inverted.

These days, the experts typically approach problems like this using Monte Carlo, often in the form of Markov Chain Monte Carlo (MCMC). One example of an analysis of this type of problem appears in section 2.4 of Pinheiro and Bates (2000) Mixed-Effects Models in S and S-Plus (Springer).

zhenglei-gao commented 10 years ago

Added an option of exactHessian=TRUE