search

ageron / handson-ml3

A series of Jupyter notebooks that walk you through the fundamentals of Machine Learning and Deep Learning in Python using Scikit-Learn, Keras and TensorFlow 2.
Apache License 2.0
7.09k stars 2.87k forks source link

[QUESTION] Chapter 2: Definition of `similarities` is subject to information leakage? #125

Open liganega opened 5 months ago

liganega commented 5 months ago

This question is referring to the jupyter notebook of Chapter 2.

===

The code below creates new 10 similarity features based on the location of the districts. But it also uses the information of "median_house_value" as sample weight.

housing_labels = strat_train_set["median_house_value"].copy()
...
similarities = cluster_simil.fit_transform(housing[["latitude", "longitude"]],
                                           sample_weight=housing_labels)

But isn't it kind of information leakage to the model? The model is going to be trained on predicting the median house value and should NOT have any direct information about it.

liganega commented 5 months ago

Using "median_house_value" as sample weight is nonsense because for prediction in the future it shouldn't be available. On the other hand, the "median_income" feature instead would be adequate for sample weight.

liganega commented 4 months ago

In fact, the sample_weight option is used only for the demonstration of how to use ClusterSimilarity class and is ignored after that. There is therefore no information leakage during the training.

However, it is still misleading to use "median_house_value" as the value for sample_weight. Using instead "median_income" results in almost the same clustering.