Gaussian Process Regression Algorithm - Githubissues

heal-research / HeuristicLab

HeuristicLab - An environment for heuristic and evolutionary optimization

https://dev.heuristiclab.com

GNU General Public License v3.0

39 stars 16 forks source link

Gaussian Process Regression Algorithm #1902

Closed HeuristicLab-Trac-Bot closed 11 years ago

HeuristicLab-Trac-Bot commented 12 years ago

Issue migrated from trac ticket # 1902

milestone: HeuristicLab 3.3.8 | component: Algorithms.DataAnalysis | priority: medium | resolution: done

2012-07-25 11:24:53: @gkronber created the issue

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-23 12:46:27: @gkronber commented

r8684: fixed a bug related to the virtual call to RecalculateResults in the constructor of DiscriminantFunctionClassificationSolution

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 12:01:38: @gkronber commented

r8732:

added caching for variances of GaussianProcessRegressionSolution.

changed hyperparameter initializers to generate initial values for hyperparameters in a larger range.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 12:03:48: @gkronber changed owner from @gkronber to @mkommend

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 12:03:48: @gkronber changed status from accepted to reviewing

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 12:56:39: @gkronber commented

r8733: added a unit test to create and run a GPR sample

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 17:07:00: @gkronber commented

r8740: removed assertion for test R² in GPR unit test as we are testing on a benchmark problem where the test set is randomly created.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-09 16:16:20: @gkronber commented

r8771: fixed a bug in the noise covariance function

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-22 12:56:29: @gkronber commented

To do: check range of all input variables and deactivate if necessary.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-22 14:38:11: @gkronber commented

r8827: fixed bugs concerning masking covariance function

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-22 16:12:39: @gkronber commented

r8829: fixed an issue that occurs in scaling if a variable is effectively constant in the training partition

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 10:30:11: @mkommend commented

Reviewing comments:

Use folders for covariance and mean functions.

Util.AttachValueChangeHandler is quite ugly

The HyperParameter class relies heavily on which method SetValue or Value.Set is called and determines according to this whether the fixed flag applies or not.

Util.SqrDist: Is the scale array always exactly as large as the number of column indixes?.

Covariance functions expose methods with column indexes which are not always forwarded to the concrete util functions (e.g., CovarianceLinear.GetCrossCovariance).

I have also tested the implementation on sample problem instances for regression and classification and did not find any bugs. However, I must admit that I didn't completely understand how the implementation works as I am lacking the basics of GP.

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 10:30:11: @mkommend changed owner from @mkommend to @gkronber

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 10:30:11: @mkommend changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 10:35:45: @gkronber commented

Replying to [comment:58 mkommend]:

Reviewing comments:

Util.AttachValueChangeHandler is quite ugly Please specify the problem in more detail. Do you mean that they should not be located in the Util class?

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 10:59:01: @mkommend commented

Replying to [comment:59 gkronber]:

Replying to [comment:58 mkommend]:

Reviewing comments:

Util.AttachValueChangeHandler is quite ugly Please specify the problem in more detail. Do you mean that they should not be located in the Util class? The "problem" is that this event registration is implemented in quite a different way as the remaining parts of HL, which makes it harder to grasp what is going on (when is the local parameter cache updated, what happens on a value change, where is the new event registered, ...).

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 11:51:01: @gkronber commented

Replying to [comment:60 mkommend]:

Replying to [comment:59 gkronber]:

Replying to [comment:58 mkommend]:

Reviewing comments:

Util.AttachValueChangeHandler is quite ugly Please specify the problem in more detail. Do you mean that they should not be located in the Util class? The "problem" is that this event registration is implemented in quite a different way as the remaining parts of HL, which makes it harder to grasp what is going on

Yes it is different to the remaining parts of HL and harder to grasp.

AttachValueChangeHandler takes a ValueParameter and registers a handler that executes the action which is specified as second parameter. Because both the value of a parameter, as well as the value of the value of the parameter can change, we have to attach to both events- In case the value of the parameter is changed we have to attach the event handler to the new value.

(when is the local parameter cache updated, what happens on a value change, where is the new event registered).

AttachValueChangeHandler does not have a local parameter cache. Thus, I believe the real issue here is the caching of parameters and values in all covariance and mean functions and not the implementation of AttachValueChangeHandler.

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 14:06:26: @gkronber commented

r8929: moved covariance and mean functions to folders

r8931: fixed incorrect handling of dimension masking in CovarianceLinear (and checked all other covariance functions)

r8932 to r8933: corrected handling of length-parameter arrays in ARD functions and prevented stacking of mask covariance functions to make sure that the length-parameter and the enumerable of selected column indexes are equally long.

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 14:40:47: @gkronber

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 14:40:47: @gkronber commented

I thought about possible changes to remove the caching of parameter values in covariance functions and came up with a possible solution returning functions with the cached value in the closure. This would remove the necessity for event registration. It also allows us to remove the SetParameters method. What do you think about this idea?
#!c#
public Func<double[,], int, int, double> GetCovarianceFunction(IEnumerable<int> columnIndices) {
      var inverseLength = inverseLengthParameter.Value.Value;
      var sf2 = scaleParameter.Value.Value;
      return (x, i, j) => {
        double d # i= j
                     ? 0.0
                     : Util.SqrDist(x, i, j, inverseLength, columnIndices);
        return sf2 * Math.Exp(-d / 2.0);
      };
    }

HeuristicLab-Trac-Bot commented 11 years ago

2012-11-20 15:01:37: @mkommend commented

Is the proposed solution really working as intended? The inverseLength and scaleParameter could change its value over time, as they are optimized during the runtime. Now I have to know at which time I have to call GetCovarianceFunction to create a new closure containing the updated parameter value, if I am even aware of this issue.

Due to this difficulties, I think the event registration fits the purpose better.

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-01 19:06:44: @gkronber changed status from assigned to reviewing

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-01 19:06:44: @gkronber changed owner from @gkronber to @mkommend

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-01 19:06:44: @gkronber commented

r8982:

removed class HyperParameter

removed the parameter value caching and event handlers for parameter caching in the implemented covariance and mean functions

the instances of mean and covariance functions configured in the GUI are now actually templates where the structure and fixed parameters of the functions can be specified. In the templates the optimized parameters are null.

Templates are used in the GaussianProcessModel to create the actual covariance and mean functions as Func objects with specified parameter values (which are optimized using LM-BFGS).

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-04 11:28:17: @mkommend changed status from reviewing to readytorelease

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-04 11:28:17: @mkommend changed owner from @mkommend to @gkronber

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-04 11:28:17: @mkommend commented

Reviewed new changesets r8929, r8931, r8932, r8933 and r8982. The refactored way of configuring covariance and mean functions is much cleaner, although I had doubts before I saw the first implementation.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-03 18:22:45: @gkronber changed status from readytorelease to assigned

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-03 18:22:52: @gkronber changed status from assigned to accepted

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-03 18:32:32: @gkronber commented

r9104: fixed a bug occurring when the masking covariance function is used.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-03 20:42:36: @gkronber commented

r9106: fixed a bug in ARD covariance functions that would occur when a parameter prior to the inverse length parameter would be fixed.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-04 00:14:51: @gkronber commented

The gradients are not returned in the same order of the parameters! This should be checked for all covariance functions.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-04 16:06:39: @gkronber commented

r9108: fixed bugs in ARD covariance functions (checked if parameter and gradient order matches for all functions)

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-04 20:50:34: @gkronber commented

r9111: fixed a calculation bug in the RQiso covariance function

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-08 16:51:32: @gkronber commented

GP should be a problem. Hyper-parameters are optimized, likelihood should be maximized.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-23 13:34:40: @abeham commented

I think I should do the same with NCA which can also be thought of as an optimization problem.

HeuristicLab-Trac-Bot commented 11 years ago

2013-02-08 21:55:04: @gkronber commented

r9211: reformulated calculation of periodic covariance function

HeuristicLab-Trac-Bot commented 11 years ago

2013-04-15 10:43:53: @gkronber commented

r9357 to r9358: minor code improvements: removed commented code, always supply non-null columnIndizes.

HeuristicLab-Trac-Bot commented 11 years ago

2013-04-15 10:43:53: @gkronber

HeuristicLab-Trac-Bot commented 11 years ago

2013-04-15 15:07:26: @gkronber commented

r9360: implemented neural network covariance function plus test case (comparison with GPML) for Gaussian processes

HeuristicLab-Trac-Bot commented 11 years ago

2013-04-15 15:09:46: @gkronber changed status from accepted to readytorelease

HeuristicLab-Trac-Bot commented 11 years ago

2013-05-10 01:37:06: @s-wagner changed status from readytorelease to closed

HeuristicLab-Trac-Bot commented 11 years ago

2013-05-10 01:37:06: @s-wagner removed resolution

Previous