Closed carusyte closed 1 week ago
multivariate distributions are not supported currently but it is something we will work to add in next few days. Would be useful if you could share the use case in more detail with a minimal example of the search space.
So excited to learn that this feature is about to be available so soon!
I'm currently experimenting a few time series models that can handle multi-variates. One of the key areas in such field is that, to my limited knowledge at this moment, the choice of significant covariates that can increase the accuracy of prediction model. Although there're wealth of means to help us make better choices, I was thinking and trying to figure out whether I could treat the selection process as hyper-parameter search and leverage Bayesian optimization for that.
I could be too naive but, basically besides "ordinary" hyper-parameters in the search space, I'm adding an np.array of floats generated by dirichlet
to rank the top-k covariates, randomly of course at the beginning. I'm expecting that bayeopt can somehow learn the relationship of this random distribution and the validation loss progressively and making better distributions that can reduce the loss, thus choosing better covariates...
This is how I specify the search space in my code:
dict(
seq_len=range(5, 300+1),
d_model=[2**w for w in range(6, 9+1)],
d_core=[2**w for w in range(5, 10+1)],
d_ff=[2**w for w in range(6, 10+1)],
e_layers=range(4, 32+1),
learning_rate=loguniform(0.0001, 0.002),
lradj=["type1", "type2", "constant", "cosine"],
patience=range(3, 10+1),
batch_size=[2**w for w in range(5, 8+1)],
dropout=uniform(0, 0.5),
activation=["relu","gelu","relu6","elu","selu","celu","leaky_relu","prelu","rrelu","glu"],
use_norm=[True, False],
optimizer=["Adam", "AdamW", "SGD"],
topk_covar=list(range(0, {max_covars}+1)),
covar_dist=dirichlet([1.0]*{max_covars}),
)
And just to get things going, I made a few change as stop gap:
https://github.com/carusyte/mango/commit/bf2386c0d331a352ea3579b16eae984ba3eb0f92
closed by #117
For example, I'm trying to use
drichlet
multivariate continuous distribution fromscipy.stats
to generate an array of random floats as one of my hyper parameters up for fine-tuning. However, if I preload historically searched hyper-parameters and pass toTuner(initial_custom=searched_hp)
, the following error will be thrown: