Make convexity validation less strict

[CallahanGuts]

153

Hi all, First i would like to thanks for this great work. I've the same trouble as already report by Galigator with smal values. I think the given answer is not totaly right, at least logically. Correct me if i'm wrong, but to my knowledge Markowitz's poretfolio take as inputs a covariances matrix of n assets (Q) and a returns vector. Since both are computed on assets prices the only assumption that can be done is that they are real numbers defined. Since data given in Galigator's test case are real values it can't be a data problem as suggest by reviewer. The raised exception suggest that given matrix is not positive semidefinite, but after trying the given data with Apache Math, Matrix calculator (www.bluebit.gr/matrix-calculator/) and MS Excel they all give the same result, which seems to discard any data problem.

After some investigation the problem is probably rooted in the eigenvalues search algorithm for small values.

Every work has some defects ! I think it will be a good think if ojAlgo teams investigated the issue to perfect their already great work.

Cheers

[apete]

The covariance matrix (that the optimisation solver calls Q) must be positive (semi)definite. In that test/example you posted it is not. It has an eigenvalue that is roughly zero and calculated to be a very small negative number. If you turn on validation it is very strict and will throw an exception even for tiny negative eigenvalues. Without validation it will most likely solve the problem anyway. (Strictly, a zero eigenvalue is not allowed.)

If I run your code I get this solution: OPTIMAL -0.7307580050276122 @ [3.6873466594435284E-17, 0.0, 0.5141624181948752, 0.3590035790996882, 0.1268340027054366]

Don't know anything about the numbers of your model, but there is a solution.

What's the defect you want fixed? Is it that the validation throws an exception for that eigenvalue?

[CallahanGuts]

Thanks Apete for your explanation, your time and reactivity. You're perfectly right, ojAlgo find the correct values. The validation mode is very strict and it's raizing exception can be a little bit misleading since a Markowitz Portfolio may very often be composed with non covariing assets (very small covariance values). May i suggest, that a more use case centric exception may be more helpfull in this case.

Regards.

[apete]

I suppose I can have a look at making the validation a little less strict, but I'd like to point out that:

1) The intention is that you only turn on validation when investigating why something has gone wrong - you don't keep it turned on. 2) Solving a Markowitz model with a few very small eigenvalues is usually not a problem. 3) The underlying optimisation algorithm does theoretically require all eigenvalues to be strictly positive. A clearly negative eigenvalue or several near-zero will cause problems. 4) Negative eigenvalues imply negative risk, and causes problems for the optimisation algorithm - make sure you don't have any. 5) If you know that your covariance matrix is "good" then it may be an idea to call that clean() method in the MarketEquilibrium class just to make sure the maths execute smoothly. If the covariance matrix has severe problems then you fix it by finding the root cause.

[CallahanGuts]

Apete, Thanks again for your time. Turning off verification and using the clean() method of MarketEquilibrium seems to cause sometime an endless loop between SparseStore.multiply() and RawStore.makezer(). Please find attached an illustrating test case. Test.java.txt

Have you any suggestion, of what can be done in this case.

Regards

[apete]

1) I don't have a problem getting a solution, but... 2) A covariance matrix like that is not supported - 2 entries on the diagonal are exactly 0.0. That just can't be! It's NOT supported! If you include more decimals then maybe you get better results.

I've relaxed the validation a bit, It will be part of ojAlgo v47 and ojAlgo-finance v2.

You need to work with your data. If you want the algorithms to work and produce good quality output, then you need to feed them good quality input.