Create execution statistics

[coveralls]

Coverage decreased (-3.2%) to 65.395% when pulling 5eade8902e16d6154964e731c72a0036c35d4b5c on abelsiqueira:feat/stats into 30dac885fc5f7096d45011758c3d52dbe2e7df68 on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-3.2%) to 65.395% when pulling 5eade8902e16d6154964e731c72a0036c35d4b5c on abelsiqueira:feat/stats into 30dac885fc5f7096d45011758c3d52dbe2e7df68 on JuliaSmoothOptimizers:master.

[abelsiqueira]

This fails 0.4 because I used the new sum syntax. Maybe we could drop 0.4 now. NLPModels already dropped, and 0.6 is scheduled for the end of February.

[coveralls]

Coverage decreased (-3.2%) to 65.395% when pulling b8839287a813710f5b4de693c4cebb22b747b6b9 on abelsiqueira:feat/stats into a9f33d555ee77aca1cf8d2fc40bcd8befa7eb789 on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-1.8%) to 66.944% when pulling b9b691bc0b09cc1da022a14dba2104d9f0d93ba0 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-1.8%) to 66.944% when pulling 793d990ec1b7348ba68498dba56219c44588ec54 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-1.8%) to 66.944% when pulling 793d990ec1b7348ba68498dba56219c44588ec54 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-1.8%) to 66.944% when pulling e98d830d10871e0375b565e326f390b3a17c7396 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+1.3%) to 70.0% when pulling 83e32dfe4686d546e1775a1a831643bf2927c99b on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+1.3%) to 70.0% when pulling 83e32dfe4686d546e1775a1a831643bf2927c99b on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+2.08%) to 70.787% when pulling 6f3130eafd2417787a0909675d4307dae3c56602 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[abelsiqueira]

@dpo, I've changed this to use status = :first_order on the solver. ExecutionStats has status as required argument, which must be one of :unknown, :first_order, :max_eval, :max_time, :neg_pred, :exception. Printing the stats, or using getStatus(stats) shows a useful message instead of the key.

This should be ready for review again.

[vepiteski]

I persist in requesting max_iter. I know the arguments against, but why not offering a user the possibility, defaulting to infinity iterations? For instance, a QP solver could well use internally c and Q defining the objective, and iterate until some precision is reached. No evaluations here. A max_iter would trap failures caused by roundoff preventing to reach the prescribed accuracy...

[coveralls]

Coverage increased (+2.3%) to 70.994% when pulling b1a24a787c925bd0422e330705488169d60a43c9 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[abelsiqueira]

I agree with @vepiteski, defaulting max_iter to infinity should not cause any harm to efficiency. I think I'm gonna refactor this now.

[dpo]

I have a hard time thinking of a situation where there wouldn't be a more meaningful measure than the number of iterations. In the QP example above, the number of products with Q, or number of factorizations of (a submatrix of) Q would be far more meaningful. Different algorithms perform different work during one iteration, making comparisons difficult, or worse, misleading.

[abelsiqueira]

We could add something related then, thinking about this case. Since ExecutionStats should be the only output on our algorithms, if a dev/user wants to create a quadratic solver measuring matrix products, then there should be a field in ExecutionStats for that (or related). Similar, but different, could be creating a new NLPModel for Quadratic Problems, and creating a new field there, though it loses the generality.

[dpo]

The matvecs are already in the counters (those should be part of ExecutionStats). Each solver could add its own stats, which could be a solver-dependent Dict, for things like number of factorizations. The number of iterations could be in there, but I don't think it should be used:

• as a measure to compare the performance of solvers;
• as a stopping condition.

[abelsiqueira]

Will add solver-specific dict.

[coveralls]

Coverage increased (+2.3%) to 70.994% when pulling 77f224ed17990e10a51aceecd7e926606c893c22 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.2%) to 71.858% when pulling b0e323400db68bbeb7ff03d84f4778bfcd653502 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.03%) to 71.739% when pulling 59cf24ba4089eddbe6edc6123195da376d427616 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage decreased (-59.8%) to 8.895% when pulling 813d062b53cc12851e16f4c03e74fbdbc2a9792f on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.7%) to 72.368% when pulling e301080beb894deb5f161d45e7194c72982cd2a8 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.6%) to 72.324% when pulling 1b5e9ce03922d98cefb2a181077cfb795a0e61bb on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.4%) to 72.135% when pulling 76a35175ff2c3f117d9399023512739b776a330d on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[dpo]

There's an error on OSX linked to gsl1. Perhaps you need to rebase this branch?

[abelsiqueira]

I think this needs a new CUTEst release, actually. Can I create CUTEst 0.2.0?

[dpo]

Sure.

[abelsiqueira]

I'll rebase this after #38.

[vepiteski]

I am in the process of testing line search algorithms. I compare the results of descent algorithms (DA) (conjugate gradient, L-BFGS, Newton) when using different linesearch routines LS. Therefore, I need that the solver outputs evaluation counts more finely grained: f is the objective, h(t)=f(x+t*d), I currently output the number of evaluations f_obj, f_grad, f_Hess, f_HV and h_obj, h_grad, h_hess to be able to know if a specific linesearch is better by computing the stepsize more cheaply (less h_evaluations) or if it is because the "quality" of the stepsize is better resulting in less f_evaluations in the DA.

[dpo]

Should LineFunctions have counters associated to them, then?

[vepiteski]

Le 2017-06-13 à 10:03, Dominique a écrit :

Should |LineFunctions| have counters associated to them, then?

That's what we have implemented. For now, we use a local copy of LineFunctions with the counters.

JPD

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/JuliaSmoothOptimizers/Optimize.jl/pull/35#issuecomment-308126206, or mute the thread https://github.com/notifications/unsubscribe-auth/AHy_GxEL5n255M6OJSJwbOD1FbgrgCMsks5sDpawgaJpZM4Lde0M.

[abelsiqueira]

You could create a subproblem NLPModels and use its counters.

LSnlp = SimpleNLPModel(t->obj(nlp, x + t*d), zeros(2), g=...)

[abelsiqueira]

Maybe change LineFunction to be an NLPModel?

[vepiteski]

Maybe this is a better solution than LineFunctions. It would also allow for bi or multi parameter searches in the vein of x+td_1+sd_2.

[abelsiqueira]

From what I can see the difference is pretty much handling Float64s instead of Arrays.

[vepiteski]

Consider a descent algo DA which uses a line search LS. How costly is it to instantiate a new LineFunction or SimpleNLPModel (either called a LSModel) at each iteration requiring a LS computation? Would it not be better to instantiate the LSModel, and then at the appropriate place (at each iteration of the DA), have a function LSModel.set!(x,d) change the current x and d?

[dpo]

Do you mean to close thisp pull request?

[vepiteski]

Maybe I'm out of topic for this pull request. My concern is cleanly allowing reporting evaluation counts from line searches separately than the main algorithm, which brings me to question the whole line search tools.

[coveralls]

Coverage increased (+3.4%) to 72.135% when pulling 76a35175ff2c3f117d9399023512739b776a330d on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.4%) to 72.135% when pulling 76a35175ff2c3f117d9399023512739b776a330d on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.4%) to 72.135% when pulling 76a35175ff2c3f117d9399023512739b776a330d on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.6%) to 72.28% when pulling f49687126cd2596008f186d09eae9f6117cf9f75 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.6%) to 72.28% when pulling f49687126cd2596008f186d09eae9f6117cf9f75 on abelsiqueira:feat/stats into 0e12c5653a233d6c315aef784bcbefe86656716f on JuliaSmoothOptimizers:master.

[dpo]

@abelsiqueira I forget where this stands.

[abelsiqueira]

This should be ready to merge after rebase.

[coveralls]

Coverage increased (+3.4%) to 72.89% when pulling f3600e5adba91fb219667fd6d1fa828f36c722f2 on abelsiqueira:feat/stats into 2bc2d180a82ecea68a9d112b393e95baa0cbd697 on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.4%) to 72.89% when pulling f3600e5adba91fb219667fd6d1fa828f36c722f2 on abelsiqueira:feat/stats into 2bc2d180a82ecea68a9d112b393e95baa0cbd697 on JuliaSmoothOptimizers:master.

[coveralls]

Coverage increased (+3.4%) to 72.89% when pulling f3600e5adba91fb219667fd6d1fa828f36c722f2 on abelsiqueira:feat/stats into 2bc2d180a82ecea68a9d112b393e95baa0cbd697 on JuliaSmoothOptimizers:master.