Incorporate Preprocessing Pipeline

[DeepBlockDeepak]

The objective is to refactor current preprocessing and feature engineering workflow to integrate custom feature engineering steps into a scikit-learn pipeline, making the codebase more modular, maintainable, and efficient.

Goals

• Encapsulate existing feature engineering logic into custom scikit-learn transformers.
• Integrate these custom transformers into preprocessing pipeline using ColumnTransformer and Pipeline.
• Ensure the refactored code maintains current functionality.

  Tasks

  1. Define Custom Transformers
• For each feature engineering function (e.g., create_is_alone, create_family_size), define a corresponding custom transformer class that extends BaseEstimator and TransformerMixin.
  • Ensure each custom transformer's transform method applies the intended feature engineering logic.
  • Implement a fit method that simply returns self, as these transformations do not require fitting.
    2. Test Custom Transformers
• Write unit tests for each custom transformer to ensure they correctly transform the data as expected.

3. Integrate Custom Transformers into Preprocessing Pipeline

• Use ColumnTransformer to apply different preprocessing strategies to numeric and categorical columns.
• Incorporate custom transformers into the pipeline with appropriate sequencing, using Pipeline for sequential transformations and * FeatureUnion if needed to combine features post-transformation.
• Ensure the final pipeline includes steps for both standard preprocessing (e.g., scaling, imputing) and custom feature engineering.

4. Update Main Workflow to Use Refactored Pipeline

• Refactor the main data processing script (preprocess_data function) to use the newly created pipeline for both training and test data preprocessing.
• Ensure that the pipeline is fitted on the training data and then used to transform both the training and test datasets.

5. Validation and Testing

• Validate the refactored pipeline against the existing workflow to ensure no loss in functionality or performance.

[DeepBlockDeepak]

Runtime Issue

During model training, encountered a ValueError indicating a type mismatch, which prevented successful model fitting:

ValueError: could not convert string to float: 'Boulos, Mrs. Joseph (Sultana)'

This error suggested a fundamental issue in preprocessing pipeline where numeric data was being inadvertently cast to object types.

Every column, including those intended to be numeric (num__*), was cast to object dtype, posing a serious problem for feeding the data into the models that expect numerical input.