This project is an approach to the Kaggle Titanic competition, aiming to predict the survival of passengers aboard the Titanic using machine learning techniques. The project involves data preprocessing, feature engineering, model training, and predicting survival outcomes. Models range from traditional models like RandomForest and SVM to custom implementations of decision trees and ensemble methods, to deep learning with PyTorch. My larger aim is to provide a comprehensive comparison of these methods by showcasing their functionalities and resulting performances.
To set up the project:
pyproject.toml
using Poetry with:
poetry install
poetry run python main.py
for training and predictions on the test set or
poetry run python user_passenger.py
for custom predictions.
See here how this project uses isort
, black
, ruff
, and unittest
for formatting, linting, and testing.
RandomForestClassifier:
The model is a RandomForestClassifier from scikit-learn, trained with 100 trees and a maximum depth of 10 (RandomForestClassifier(n_estimators=100, max_depth=10, random_state=1)
). The model is trained in src/kaggle_titanic/random_forest/rf_main.py
and persisted to models/rf_titanic_model.pkl
. (Accuracy: 0.85
)
Hand-Rolled Decision Tree Classifier: A custom decision tree model is implemented from scratch, showcasing a more manual approach to classification. The model constructs a decision tree using Gini Impurity and Information Gain as metrics for optimal feature selection and splitting. The tree consists of Leaf
and Internal_Node
classes to handle predictions. The decision-making process traverses the tree, making decisions at each node based on the data until it reaches a leaf node. The model is detailed in src/kaggle_titanic/decision_tree/decision_tree.py
and demonstrates comparable performance to the SVM model (Accuracy: 0.72
). See Math Notes Wiki for split-algorithm details.
SVM (Support Vector Machine): The project incorporates a Support Vector Machine model using the SVC class from scikit-learn. The SVM model is configured with an RBF kernel (kernel='rbf'
), a scale gamma value (gamma='scale'
), and a regularization parameter C set to 1 (C=1
). This model is particularly effective for high-dimensional spaces and is implemented in src/kaggle_titanic/svm/svm_train.py
. It is trained and evaluated in a similar fashion to the RandomForestClassifier. (Accuracy: 0.73
)
Hand-Rolled RandomForestClassifier: An extension of the hand-rolled Decision Tree Classifier, this model implements the RandomForest algorithm from scratch. It constructs multiple decision trees on bootstrapped samples of the dataset and uses majority voting for classification. This approach demonstrates the power of ensemble learning in improving prediction accuracy and robustness over individual decision trees. The implementation can be found in src/kaggle_titanic/rfc_hand_rolled/random_forest.py
. (Accuracy: Reaches as high as 0.87)
Naive Bayes: This model uses GaussianNB
scikit-learn, a probabilistic classifier based on applying Bayes' theorem with the assumption of independence between features. The training and evaluation process is handled in src/kaggle_titanic/naive_bayes/bayes_main.py
.
PyTorch Binary Classifier: A PyTorch Neural Network model, designed for binary classification tasks. The network architecture consists of an input layer adapted to the dataset's feature size, a hidden layer with 64 neurons, ReLU activation for introducing non-linearity, dropout for regularization, and a sigmoid output layer for probability prediction. The structure and training process are in src/kaggle_titanic/pytorch/pytorch_binary.py
.
The project features three distinct models: RandomForestClassifier, SVM, and a hand-rolled Decision Tree Classifier. Each model has been evaluated for its accuracy:
Model Accuracy Comparison: A comparative view of the accuracy scores achieved by each model.
Feature Importances: This bar chart ranks the features by their importance in the RandomForestClassifier model. The length of the bar represents the feature's weight in the model, with
Title_Mr
, Fare
, and Age
being among the most influential for predicting survival on the Titanic. Notably, Title_Mr
emerges as a significant predictor — a result of extracting titles from passenger names and applying one-hot encoding during the preprocessing phase, as defined in src/kaggle_titanic/features.py's extract_title()
.
Confusion Matrix: (0 for not survived, 1 for survived). The y-axis represents the actual labels.
ROC Curve: Evaluating Model's Diagnostic Ability
Histogram of Predicted Survival Probabilities
The project utilizes the Titanic dataset from Kaggle, obtained via kaggle competitions download -c titanic
. From Kaggle:
train.csv
contains the details of a subset of the passengers on board (891 to be exact) and importantly, will reveal whether they survived or not, also known as the "ground truth".- The
test.csv
dataset contains similar information but does not disclose the "ground truth" for each passenger. It's your job to predict these outcomes.gender_submission.csv
: The expected submission format.
The project is structured to provide a comprehensive approach to the Titanic survival prediction task. Key components and their functionalities are as follows:
main.py
: Central entry point. It handles data loading, preprocessing, model training, evaluation, and prediction generation. The script is versatile, allowing users to select between RandomForestClassifier, SVM, Decision Tree, or to compare all models by using the --model
argument. Outputs a ./submission.csv
file for the test set predictions for the Kaggle submission.Example Usage:
poetry run python main.py --model random_forest
poetry run python main.py --model decision_tree
poetry run python main.py --model svm
poetry run python main.py --model custom_rfc
poetry run python main.py --model naive_bayes
poetry run python main.py --model pytorch
poetry run python main.py # defaults to random_forest
rf_hyperparameter_tuning.py
: This script utilizes GridSearchCV from scikit-learn to exhaustively search through a predefined grid of hyperparameters and find the combination that yields the best performance. The script processes the training data, splits it for validation, and applies a grid search on the RandomForestClassifier. Interestingly, this script doesn't not produce a more accurate model than that provided with default values inside of Random Forest Main Script
user_passenger.py
: Takes user input for a single passenger and outputs the survival probability.
Please see CONTRIBUTING.md for guidelines on how to contribute, set up your environment, run tests, and more.
This project is an initial implementation of the Titanic survival prediction. Ideally, this model will someday achieve a perfect accuracy score!
For more detailed information about the scripts and model training, please refer to the source code within the src/kaggle_titanic
package.