Chapter 2 (page90), full_pipeline.transform()

[luluqinqin]

Hi Aurélien,

Thank you for a GREAT book! I really enjoy reading it and practicing the examples in the book. In chapter 2, when we need to transform the test data, you mention that not using the fit_transform() as we don't want to fit the test data, and we should use the transform() function. But the SimpleImputer needs to run the fit() first before running the transform(). Can you please illustrate a solution for that? Thank you!!

Lu

Code:

num_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy = 'median')),
    ('std_scaler', StandardScaler()),
])

#
num_attr = list(test_num)
cat_attr = ['ocean_proximity']   
full_pipeline = ColumnTransformer([
        ("num", num_pipeline, num_attr),
        ("cat", OneHotEncoder(), cat_attr),
    ])
test_x_tr = full_pipeline.transform(test_x) 
test_x_tr

Error:

NotFittedError: This ColumnTransformer instance is not fitted yet. Call 'fit' with
appropriate arguments before using this estimator.

[ageron]

Hi @luluqinqin ,

Thanks for your kind words! 😊 And sorry for the late reply...

You are right that you must always call fit() on a transformer before you can call transform() (or of course you can call fit_transform(), which does both). Similarly, you must always call fit() on a predictor before you call predict().

BUT, the call to fit() must always be done on the training set. Once that's done, you can call transform() (or predict(), in the case of a predictor) on the training set or any other set.

So, your code would be correct if we replaced the last two lines with this code:

train_x_tr = full_pipeline.fit_transform(train_x) 
test_x_tr = full_pipeline.transform(test_x)

Hope this helps.