search

stair-lab / CT-Disease-Detection

The STAIR Lab Repository for the CT Disease Detection Project
MIT License
0 stars 0 forks source link

Debugging Model Performance on 1 Continuous Target: CTBiomarkers.CalciumScoring.AbdominalAgatston #1

Open Ahmed14974 opened 1 month ago

Ahmed14974 commented 1 month ago

Previous comments:

Ahmed14974: As previously observed through preliminary modeling iterations, any and all models trained on the input images to predict the Continuous Target variable CTBiomarkers.CalciumScoring.AbdominalAgatston have not shown any degree of decent performance. This Issue will document all approaches to be considered for the purposes of tuning and improving model performance, including model iterations which have already been attempted

echen4096: I tested a ResNet model with CoordConv layers and pretrained weights (commit). However, model performances remain poor.

Results:

mse: 0.013112680986523628,
r_squared: -0.013185453842119443,
explained_variance: -0.010275483131408691

mse: 0.01319037564098835,
r_squared: -0.019188805176170165,
explained_variance: -0.004194855690002441
Ahmed14974 commented 1 month ago

Experiments Attempted for CTBiomarkers.CalciumScoring.AbdominalAgatston

Details of models attempted with architecture specifics and results are presented below. For all models, the target variable is min-max rescaled to a range of [0,1].

Exp_name | View | Learning Rate | Head Attached | Sigmoid Layer Appended to Head | model only_last_layer | Image Transformations | Num Epochs | Test MSE | Test R-squared | Test Explained Variance Score -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- mahmedc_resnet34_1e-3_two_view | two_view | 1e-3 | No | No | FALSE | legacy | 10 | 0.013425091 | -0.0373247 | -0.015828609 mahmedc_resnet34_1e-3_one_view | one_view | 1e-3 | No | No | FALSE | legacy | 10 | 0.013014367 | -0.0055890 | -0.000684977 mahmedc_resnet34_1e-3_one_view_head_model | one_view | 1e-3 | Yes | No | FALSE | legacy | 10 | 0.013009196 | -0.0051894 | -0.004006982 mahmedc_resnet34_1e-3_one_view_head_model_sigmoid | one_view | 1e-3 | Yes | Yes | FALSE | legacy | 10 | 0.013236634 | -0.0227629 | -0.000447869 mahmedc_resnet34_1e-3_one_view_head_model_sigmoid_onlylast | one_view | 1e-3 | Yes | Yes | TRUE | legacy | 10 | 0.013681340 | -0.0571244 | -0.024955988 mahmedc_resnet34_1e-3_one_view_head_model_sigmoid_onlylast_transforms | one_view | 1e-3 | Yes | Yes | TRUE | alligned with ResNet pre-training | 10 | 0.013806610 | -0.0668038 | -0.013790250 mahmedc_resnet34_1e-4_one_view_head_model_sigmoid_onlylast_transforms_50 | one_view | 1e-4 | Yes | Yes | TRUE | alligned with ResNet pre-training | 50 | 0.013779610 | -0.0647176 | -0.002306342 mahmedc_resnet34_1e-5_one_view_head_model_sigmoid_onlylast_transforms_50 | one_view | 1e-5 | Yes | Yes | TRUE | alligned with ResNet pre-training | 50 | 0.049071338 | -2.7916246 | -0.243152857

Training and Validation Loss Curves (MSE) for the model with lowest Test MSE from above

image image

Exp_name | View | Learning Rate | Head Attached | Sigmoid Layer Appended to Head | model only_last_layer | Image Transformations | Num Epochs | Test MSE | Test R-squared | Test Explained Variance Score -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- mahmedc_resnet101_1e-4_one_view_head_model_sigmoid_transforms_50_batch16 | one_view | 1e-4 | Yes | Yes | FALSE | alligned with ResNet pre-training | 50 | 0.015156988 | -0.171144077 | 0

Head attached is as follows (Sigmoid appended as the last layer, not present in some experiments)

class Head(nn.Module):
    def __init__(self, in_planes=1024, mid_planes=512, out_planes=35):
        super().__init__()

        self.layers = nn.Sequential(
            #AdaptiveConcatPool2d(sz=(1, 1)),
            #nn.Flatten(),
            nn.BatchNorm1d(in_planes, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
            nn.Dropout(p=0.25, inplace=False),
            nn.Linear(in_features=in_planes, out_features=mid_planes, bias=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm1d(mid_planes, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
            nn.Dropout(p=0.5, inplace=False),
            nn.Linear(in_features=mid_planes, out_features=out_planes, bias=True),
            nn.Sigmoid())

    def forward(self, x):
        return self.layers(x)

Image Transforms before and after alignment with ResNet pre-training are as follows

## Old Transforms
train_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(p=0.2),
        transforms.RandomApply([
            transforms.RandomAffine(10),
            transforms.RandomResizedCrop(256, scale=(1.0, 1.1), ratio=(0.75, 1.33)),
            transforms.ColorJitter(brightness=0.2, contrast=0.2)
        ], p=0.75),
        transforms.RandomPerspective(distortion_scale=0.2, p=0.75),
        transforms.ToTensor(),
        transforms.Normalize((0.55001191,), (0.18854326,))
    ])

## Aligned Transforms
train_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.Grayscale(num_output_channels=3),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])