search

SheffieldML / GPyOpt

Gaussian Process Optimization using GPy
BSD 3-Clause "New" or "Revised" License
927 stars 261 forks source link

Usage question: high nD space, best way to define bounds #210

Open gunshi opened 5 years ago

gunshi commented 5 years ago

Hi, I'm using gpyopt to optimise in the latent space of an autoencoder (i have a feature vector Xi of dimension 50, and i have a way of giving a real number 'score' to the output when this feature vector is decoded through the decoder of the autoencoder of a neural network, so my objective function is -> the decoding + scoring). Now if i want to add domain bounds for these 50 dimensional vectors (say element-wise max and min for all the 50 dimensional vectors Xi in my starting training set X, so that it samples only between the extreme values for each dimension, already seen in the set), is there any way to achieve that? I was not able to find an existing way to do it from my reading of the documentation and examples, but maybe I could do it be extending a class? EDIT: I mean a way to define bounds besides adding them explicitly to a 'domain' variable for each position in the 50 dimensions Thanks! Gunshi

gunshi commented 5 years ago

I did find another way: https://github.com/SheffieldML/GPyOpt/blob/9365794b4dbef9f32b35437ec4e7e9ca5c9442eb/GPyOpt/objective_examples/experimentsNd.py#L17

passing in a list of tuples(as done on the above link's line 17) for each of the 50 components. Closing since this seems to be sufficient. Please let me know however if there is a better way Thanks.

gunshi commented 5 years ago

Hi, I seem to be getting memory errors when optimizing over 50 dimensional vectors with GP/sparseGP and Expected Improvement as the aquisition function, all variables being continuous (can post a minimal python notebook and data file here) but first I wanted to inquire if this is expected behaviour in the sense of the dimensionality being too high? Thanks Gunshi

gunshi commented 5 years ago

errors in question:

with GP:

Traceback (most recent call last): File "run_bo2.py", line 102, in x_next = bo_step.suggest_next_locations() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/core/bo.py", line 67, in suggest_next_locations self._update_model(self.normalization_type) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/core/bo.py", line 251, in _update_model self.model.updateModel(X_inmodel, Y_inmodel, None, None) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/models/gpmodel.py", line 81, in updateModel self._create_model(X_all, Y_all) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/models/gpmodel.py", line 65, in _create_model self.model = GPy.models.GPRegression(X, Y, kernel=kern, noise_var=noise_var) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/parameterized.py", line 58, in call self.initialize_parameter() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 337, in initialize_parameter self.trigger_update() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/updateable.py", line 79, in trigger_update self._trigger_params_changed(trigger_parent) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 134, in _trigger_params_changed self.notify_observers(None, None if trigger_parent else -np.inf) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/observable.py", line 91, in notifyobservers [callble(self, which=which) for , , callble in self.observers] File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/observable.py", line 91, in [callble(self, which=which) for , _, callble in self.observers] File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 508, in _parameters_changed_notification self.parameters_changed() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/core/gp.py", line 267, in parameters_changed self.posterior, self._log_marginal_likelihood, self.grad_dict = self.inference_method.inference(self.kern, self.X, self.likelihood, self.Y_normalized, self.mean_function, self.Y_metadata) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/inference/latent_function_inference/exact_gaussian_inference.py", line 53, in inference K = kern.K(X) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/kern/src/kernel_slice_operations.py", line 86, in wrap ret = f(self, s.X, s.X2, *a, kw) File "", line 2, in K File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/caching.py", line 283, in g return cacher(*args, *kw) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/caching.py", line 179, in call new_output = self.operation(args, kw) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/kern/src/stationary.py", line 114, in K r = self._scaled_dist(X, X2) File "", line 2, in _scaled_dist File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/caching.py", line 283, in g return cacher(*args, *kw) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/caching.py", line 179, in call new_output = self.operation(args, kw) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/kern/src/stationary.py", line 168, in _scaled_dist return self._unscaled_dist(X, X2)/self.lengthscale File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/kern/src/stationary.py", line 138, in _unscaled_dist r2 = -2.tdot(X) + (Xsq[:,None] + Xsq[None,:]) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/util/linalg.py", line 323, in tdot return tdot_blas(args, kwargs) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/util/linalg.py", line 305, in tdot_blas out = np.zeros((nn, nn))

error with sparseGP:

Traceback (most recent call last): File "run_bo2.py", line 102, in x_next = bo_step.suggest_next_locations() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/core/bo.py", line 67, in suggest_next_locations self._update_model(self.normalization_type) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/core/bo.py", line 251, in _update_model self.model.updateModel(X_inmodel, Y_inmodel, None, None) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/models/gpmodel.py", line 81, in updateModel self._create_model(X_all, Y_all) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPyOpt/models/gpmodel.py", line 67, in _create_model self.model = GPy.models.SparseGPRegression(X, Y, kernel=kern, num_inducing=self.num_inducing) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/parameterized.py", line 58, in call self.initialize_parameter() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 337, in initialize_parameter self.trigger_update() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/updateable.py", line 79, in trigger_update self._trigger_params_changed(trigger_parent) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 134, in _trigger_params_changed self.notify_observers(None, None if trigger_parent else -np.inf) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/observable.py", line 91, in notifyobservers [callble(self, which=which) for , , callble in self.observers] File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/observable.py", line 91, in [callble(self, which=which) for , _, callble in self.observers] File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/paramz/core/parameter_core.py", line 508, in _parameters_changed_notification self.parameters_changed() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/models/sparse_gp_regression.py", line 66, in parameters_changed super(SparseGPRegression, self).parameters_changed() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/core/sparse_gp_mpi.py", line 122, in parameters_changed super(SparseGP_MPI,self).parameters_changed() File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/core/sparse_gp.py", line 80, in parameters_changed mean_function=self.mean_function) File "/u/guptagun/anaconda3/envs/rdkit/lib/python3.6/site-packages/GPy/inference/latent_function_inference/var_dtc.py", line 145, in inference dL_dm = -np.dot((_LBi_Lmi_psi1.T.dot(_LBi_Lmi_psi1)) MemoryError

Thanks Gunshi