Closed tomblaze closed 5 years ago
Hey @tomblaze ,
CCMPred uses the Conjugate Gradient method for optimization. It iteratively computes the partial derivatives w.r.t. each of the parameters and estimates the step width (i.e. how far it goes in the direction of the gradient) using some approximation instead of the exact width (which would require to compute the Hessian). This approximation depends on the last steps which the algorithm did and develops continuously.
If you load the parameters from a previous run, the parameters of the Conjugate Gradients are not set to the last state but initialized again. It also does not have access to all the previous steps. Since you already are in a local optimum after the first run, the initial parameters of the Conjugate Gradient are probably way to "big", which means instead of getting closer to the previous optimum, it over steps an slowly optimizes towards another optimum (or back to the first one, i don't remeber whether the pseudo likelihood function is unimodal).
This all means that to continue from where you left off after the first run, you would also have to save all the parameters from the Conjugate Gradient and set them at initialization too. We implemented Conjugate Gradients in libconjugrade, so maybe you want to take a look at that library. Look for parameters like gprevnorm or prevalpha which are used by the next iteration step in conjugrad.c.
I hope this helps.
Greets, Markus
@grubermar @tomblaze, I think there is a bug in the read_raw method. I submitted a PR, but won't have the time to verify the fix today.
Thanks. I imagine you'll want to fully double-check the change and you can take your time doing that but making the change locally does make the output more sane on re-initialization.
First pass:
iter eval f(x) ║x║ ║g║ step
1 1 171122 13287.4 77283600 9.23e-05
2 1 166286 13285.7 57418396 7.9e-05
3 1 161400 13287.1 47159776 7.96e-05
...
36 2 92304.7 17349.2 357860.28 0.000339
37 3 92271.6 17316.8 326918.62 0.000141
Done with status code 0 - Success!
Second pass:
iter eval f(x) ║x║ ║g║ step
1 2 92130.3 17267.3 306641.62 0.000922
2 1 92028.7 17243.6 298147.44 0.000564
3 1 91928.7 17213.1 234786.17 0.000518
4 1 91840.1 17174.3 256977.78 0.000635
5 2 91786.8 17145.4 76054.055 0.000348
Done with status code 0 - Success!
@tomblaze, I just merged the fix into master. Thanks for pointing it out 👍
I am interested in using the MRF parameters and wanted to do a sanity-check comparing values from your code and my own. I believe the raw files should have this information.
One thing I'm not understanding is how the initialization option works. My expectation would be that if I train a model, save the "raw" parameters, and then re-run the code with option "-i" set to the same raw file, re-starting the training should have the f(x) scores initialized at the same place.
However even on example input this doesn't seem to be the case. See below, first I run:
Then I run...
It seems that the
||x||
variable is related between the two runs but it seems odd to me that thef(x)
value is markedly higher than where it ends up in first round of optimization and that ultimately optimization from this initialization takes longer and ends up at a worse optimum. Perhaps I am misunderstanding something but for now this seems counter-intuitive and I'd appreciate any clarification you could give.Thanks!