Open koaning opened 3 weeks ago
If this is out of scope for this project though I will gladly hear it.
Implementing CV isn't necessarily out of scope, but IbisML will probably need to reimplement it, because reproducible shuffling/splitting on databases is hard-ish. Here's the implementation of the much-simpler train_test_split
: https://github.com/jitingxu1/ibis-ml/blob/main/ibis_ml/utils/_split.py
CV was something we intentionally deprioritized for initial release, but will definitely take a second look!
Hi @koaning Thanks for your input.
The reasons we deprioritized it for our initial work is that, I copied my reply in https://github.com/ibis-project/ibis-ml/issues/135#issuecomment-2308956315:
We still could do the model hyperparameter's gridsearchCV based on IbisML's output, but the ibisML recipe should not be included in the sklearn pipeline, since IbisML have no cv splitter
right now
IbisML do the feature transformation on a compute backend, it will transfer the preprocessed data from compute backend to the training process, the data movement cost will be significantly high for training with large datasets.
In my limited and outdated modeling experience, random or grid search may not be suitable for tuning some preprocessing transformations. Instead, preprocessing parameter tuning often requires feature analysis, for example, choosing between imputing a feature with the median or mean. Do you have some insights here. we would like to learn to see if we could prioritize this work.
It can be pretty reasonable to ask questions like "what if we add this feature, how much uplift do we get?". If you ever want to do stuff like that ... it would sure help to be able to hyperparam that.
It's not unreasonable given the current design not to allow for this though. Feels like fair game considering that it is a different backend.
It can be pretty reasonable to ask questions like "what if we add this feature, how much uplift do we get?". If you ever want to do stuff like that ... it would sure help to be able to hyperparam that.
I agree, this is a very fundamental part of the workflow.
It's not unreasonable given the current design not to allow for this though. Feels like fair game considering that it is a different backend.
Part of the benefit of Ibis (and, by extension, IbisML) is that you can scale by choosing a more appropriate backend, instead of changing code. The way I see it (actually, the way I'm working on a large dataset myself), it would be a very reasonable user workflow to want to experiment locally on a smaller using the DuckDB backend, then scale on the full dataset using a distributed backend. As a result, I want to be able to use IbisML to define my Recipe
, use it in a scikit-learn pipeline, and tune hyperparameters on smaller data.
I never planned to tune hyperparameters on the full, multi-TB dataset (what's the point?), but I do want to run the same Recipe
/pipeline with the established hyperparameter values.
It makes sense to support this, but I wanted to highlight a few potential scenarios to consider.
Did some investigation this morning:
To enable this, we need to implement a scikit-learn compatible splitter in IbisML. In scikit-learn, the splitter returns indices for each split, and the dataset is materialized during the fit process. Currently, scikit-learn supports pandas, numpy, polars,
and list_index. I suspect the _list_index
with key_type == bool
could work with Ibis tables, but this requires further investigation.
Another approach could be enabling scikit-learn to directly support ibis Table
.
Lastly, we should work on making the get_param and set_param functions more granular.
We may not have the bandwidth to tackle this in the next 2 weeks, but
it’s something we plan to address soon.
I guess there is also another thing to consider: isn't it the goal of IbisML to not just support scikit-learn but also other frameworks? I am mainly mentioning this because there is a risk of overfitting to sklearn too.
I guess there is also another thing to consider: isn't it the goal of IbisML to not just support scikit-learn but also other frameworks? I am mainly mentioning this because there is a risk of overfitting to sklearn too.
It's a bit of both. :) As you say, we don't want to (and possibly can't) overfit to scikit-learn. For example, re some of the concerns brought up in https://github.com/ibis-project/ibis-ml/issues/135#issuecomment-2307729193—we probably don't need to overly focus on enabling things like caching. Especially as users look to scale, the right approach with IbisML may involve adding a bit of simple custom code or something. For hyperparameter tuning, from some limited research when the IbisML project was started, it sounded like a lot of the target users in industry were using things like Optuna, so we may focus more on enabling that workflow.
That said, scikit-learn is the most popular way for Python-oriented data scientists to work, and there's an extent of wanting to meet users where they are/fit into existing workflows where it's reasonable.
To that end, I can probably look into at least:
get_param
/set_param
on stepslet's start with sklearn and consider other framework during design and implementation if possible. Thank you.
pytorch and tensorflow itself does not support kfold cv, some high level lib may do this.
I may have found a bug related to how IbisML integrates with grid search from sklearn, but it could be that this is out of scope of the project.
This gave me this error:
It seems that the issues originates from the Ibis side of things, hence the ping. If this is out of scope for this project though I will gladly hear it.