Open momegas opened 1 year ago
Do you believe that there is a time threshold (e.g. not take more than 20 seconds)?
Do you believe that we have to set an evaluation metric threshold(e.g. if model achieves 90% accuracy, pick that model)?
The model training will be performed once per training set. Means that we will retrain the model only if the training set changes. Do we need to keep track of models (e.g. by using MLflow?)
Regardless of whether we use MLflow or not, do we need to save somewhere the hyper-parameters of the optimal model?
I think its important to keep the target of Whitebox in mind. The target is monitoring not create models (at least not now) With this in mind, I think that we should either have a quick tuning or not at all. How I understood this issue was that it would be just some adjustments on the training. Not create a full other feature.
Think about this and if we can have just this in the timebox we have good. Otherwise, I would look at something else.
Having some discussions with @stavrostheocharis , we concluded that the requirements of this task are still pretty blurry. I will try to simplify them with some simple questions below, so please @momegas - when you have the time, let us know.
I think we should not spend more time on this as a better model will not give much value to WB at the moment since we are missing more core features. Feel free to close this if needed @NickNtamp
Sure I can close the ticket @momegas . Before do it, I want to remind to both you and @stavrostheocharis that by not exploring combinations in order to increase the possibility of building a better model in an unknown dataset we accept the high risk of explaining a trash-model. Just imagine that we could build a model that has an accuracy of 20% and we will use it for our explainability feature.
I would keep this as an issue in the backlog, in order to further investigate it and implement an enhancement in the future
It was actually requested! You are right. I will re-open this.
We should explore alternatives like https://optuna.org/ here.
The hyper-parameters tuning is pretty easy to be performed using Gridsearch.
There are some questions thought:
cc: @momegas , @gcharis , @stavrostheocharis