search

automl / Auto-PyTorch

Automatic architecture search and hyperparameter optimization for PyTorch
Apache License 2.0
2.37k stars 287 forks source link

Do you have any particular reason to require at least 2 input features? #56

Closed maxmarketit closed 3 years ago

maxmarketit commented 4 years ago

I found that the mininumu number of features for input is 2. Is there any reason?

Because I made custom loss function utilizing two input variables asymmetrically,

and it does not seem to work. So I am wondering if there is any particular reason

that might lead to poor training... I might add my particular loss function and the result later

LMZimmer commented 4 years ago

Could you specify what you are trying to do? Which data are you working with, with search space etc.?

maxmarketit commented 4 years ago

It seems my loss function works as it is expected, but the result is not up to my expectation so you might take a look at it and see if there is anything wrong with the optimization.

I will introduce my objective briefly. I am doing quantile regression with q-x as input and y(q) as output. But since loss is something like max(q(y-t), (q-1)(y-t)), we need to know q in calculating loss, I would let q have direct path to loss when designing the architecture. But I dont know that option when using automl, I put q in target and made modification to the loss, which is QuantileLoss2 in my code.

Anyhow, my loss works fine as you can see Quantile-QinputLoss-Copy1.html.

As I am working this example as a toy example, checking if auto-pytorch suits my problem, the result of AutoPyTorch-Quantile-qinput.html turns out a bit of disappointment, considering all the time it spent searching ~ 10 hrs I guess.

So the designer and developer of auto-pytorch might see the possible progess somewhere somehow I guess.

Anyway I always thank you for your dedication and commitment!

Since I dont know how to attach a file here, Here's the link to the github

AutoPyTorch-Quantile-qinput.ipynb Quantile-QinputLoss-Copy1.ipynb

I am also wondering what would be the best options for autonet.fit. Any suggestion would be welcome.

I guess min_budget should not be big because it would take away chances of trying many models and I gues trying as many model as possible is crucial given huge hyper-parameter space.

maxmarketit commented 4 years ago

I think the problem is related to this issue

Thanks to your suggestions in this issue, I succeeded to change the loss but I do not know how to change the scoring metric or the loss for BOHB. Could you explain how to change the scoring metrics and the loss for BOHB? Thank you in advance!

ravinkohli commented 3 years ago

Hey, thanks for this question. Actually, we have been working on a new version of autopytorch that is available on the branch `development, for docs, see this. Now we feel it's ready for people to test their workflows with this version.

So I suggest you to try out your data with the development version. For an example on how to use the API, find it here

You can specify the metric to be optimized using the optimize_metric argument to the search function of the task. Feel free to ask any questions about the implementation here. And if you feel the issue is resolved, kindly close it.