Open Vincent-Maladiere opened 1 year ago
I'm working on adding the key
argument, and the MultiAggJoiner
:)
Hey! Would having Bayesian means be useful? It seems to me the spirit of skrub is to provide good defaults to users. One thing that can happen with AggTarget and AggJoiner out of the box is to do aggregates on small groups, which can lead to overfitting
shrinking towards the overall aggregate across groups sounds like a useful option to add
I guess it only applies for some of the aggregation operations
I guess it only applies for some of the aggregation operations
My initial understand of skrub is that it should provide good defaults to users. It's nice to have AggJoiner and AggTarget be able to compute a variety of statistics. But in practice (e.g. Kaggle) it's more or less sufficient to compute the mean. So if these classes are going to be used as part of TableVectorizer, maybe the default could be a mean that shrinks towards the overall mean. I think this would be a good pit of success for most users.
Hi @MaxHalford, that sounds like a good idea, and something that is already performed in scikit-learn TargetEncoder. We should create a small benchmark to explore this idea in AggJoiner and AggTarget.
I'd be interested to work on screening when I'm done with the multi joiners. I think it's a feature that might also be useful for other estimators than the Joiners, if you think that's the case we can open a new issue on this topic
as discussed with @TheooJ , the AggTarget does not implement cross-fitting (see eg the target encoder doc) which can cause serious overfitting of the downstream estimator. moreover shrinking/smoothing is probably important when there are some values with few matches in the joining column
This meta-issue is a roadmap for the developments of the
AggJoiner
andAggTarget
estimators, currently implemented in #600.We need to merge the following PRs before tackling any of the developments below.
This is not a personal roadmap. Anyone is welcome to contribute! 🙂
key
argument As discussed in https://github.com/skrub-data/skrub/discussions/751Joiner
andAggJoiner
currently accept multiple aux tables, but this creates complexity because some parameters can be iterable of iterable of str. Instead, we need to createMulti{AggJoiner, Joiner}
to handle multi tables, and only accept a single table otherwise.count
from the other operations The count operation is currently applied on all categorical columns, which create duplicates. Also, we want to use it even if we only have numerical columns. So, we need to treat this operation separately.TableVectorizer
'scolumn_specific_transformers
.polars.LazyFrame
Lazyframes don't have the__dataframe__
attribute. A duck-typing likelazy_df.__dataframe__ = None
is enough to get it working, but we might want a less hacky solution.value_counts
andhist
operations for Polarsfuzzy_join
function or composing theJoiner
class.Any other ideas?