guillaume-chevalier
commented
3 years ago Suggested Algo:
## Algo :
svd_encode = SVDEncoder(featurizer=Linearizer())
encoded_good_hp = svd_encode.fit_transform(good_hyperparams)
encoded_bad_hp = svd_encode.transform(bad_hp)
Good_posterior = self._create_posterior(encoded_good_hp )
bad_posterior = self._create_posterior(encoded_bad_hp )
## Classes :
Class Linearizer():
Def featurize():
#apply exp on log feature
#apply log on exp feature
#...
Def unfeaturize()
# ...
Class SVDEncoder():
Def fit()
self._re_encode_choices()
self._featurize_relinearize_dims()
self.featurize()
self.fit_svd()
Def transform() # encode()
self.featurize()
self.svd()
Def inverse_transform() # decode()
self.undo_svd()
self.undo_featurize()
self._validate_clip_encodings(self.original_space_dict)
Def fixed_point_iterations(b, a, N_iters)
Transformed_a = a
For _ in range n_iters:
bb = self.transform(a)
aa = a + (b - bb)/2 # find midpoint between a and b in encoded space
a = self.inverse_transform(aa)
a = self._validate_clip_encodings(a, self.original_space_dict)
If is_better(a, transformed_a):
transformed_a = a
Return transformed_a 
stale[bot]
Is your feature request related to a problem? Please describe. Hyperparameters are sometime inversely correlated in score (e.g.: learning rate v.s. batch size).
Describe the solution you'd like An SVD decomposition of dimensions of the hyperparameter space within the TPE optimization so as to optimize each dimension as independently as possible within this linear transformation of the space.