Closed lsch0lz closed 2 years ago
Hi, what is the line that crashes ? Can you paste a code snippet of the actual code that crashes ? Is the crash when using gradcam or when doing something else ?
Hi! It crashes when using the cam object:
AxisError Traceback (most recent call last)
[<ipython-input-7-db02c72a6749>](https://vl7u1jefvqc-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220518-060055-RC00_449411064#) in <module>()
40
41 # You can also pass aug_smooth=True and eigen_smooth=True, to apply smoothing.
---> 42 grayscale_cam = cam(input_tensor=input_tensor)
43
44 # In this example grayscale_cam has only one image in the batch:
7 frames
<__array_function__ internals> in mean(*args, **kwargs)
[/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py](https://vl7u1jefvqc-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220518-060055-RC00_449411064#) in _count_reduce_items(arr, axis, keepdims, where)
74 items = nt.intp(1)
75 for ax in axis:
---> 76 items *= arr.shape[mu.normalize_axis_index(ax, arr.ndim)]
77 else:
78 # TODO: Optimize case when `where` is broadcast along a non-reduction
AxisError: axis 2 is out of bounds for array of dimension 0
The input_tensor is torch.Size([1, 3, 100, 100])
Thanks. What is the definition of this feature extraction network that makes it crash, and of the target_layer ? The output of the feature extraction network is still the number of categories, right ?
@jacobgil The last Layer is defined as:
model_ft.fc = nn.Linear(num_ftrs, num_classes)
num_classes is 2, because I want to classify the classs 'doctor' and 'nurse'. So yes, the output is the number of categories.
eval()
gives me the following structure:
ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=2, bias=True)
)
Does that information helps?
Yes but I'm still missing some context, sorry. What is the difference in the model/setting when it works and when it crashes ?
@jacobgil no worries! I'm glad you are helping. So if feature_extract
is true the model only learns the weights for the parameters that don't require gradients. As far as I understand this tutorial Fine tune PyTorch Model only the last layer is changed because of this.
When I compare the structure between feature_extract
True and False it seems the same. But GradCam only works when feature_extract
is false. I could image that the change of the last layer is messing something up, but I'm not sure tbh
Maybe I'm missing something here...
Oh now I get it. It might be possible that since the require grads is false for that model, grad-cam isn't able to compute the gradients for it. Can you try turning on requires_grad for the target layer in the model before calling the cam object (and then if you need to, turn it off when it's finished) ?
Makes sense. Do you mean turning on requires_grad like this:
model.layer4[-1].requires_grad = True
target_layers = [model.layer4[-1]]
This still gives me the following error:
AxisError Traceback (most recent call last)
[<ipython-input-8-72acee2d8440>](https://vl7u1jefvqc-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220518-060055-RC00_449411064#) in <module>()
42
43 # You can also pass aug_smooth=True and eigen_smooth=True, to apply smoothing.
---> 44 grayscale_cam = cam(input_tensor=input_tensor)
45
46 # In this example grayscale_cam has only one image in the batch:
7 frames
<__array_function__ internals> in mean(*args, **kwargs)
[/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py](https://vl7u1jefvqc-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220518-060055-RC00_449411064#) in _count_reduce_items(arr, axis, keepdims, where)
74 items = nt.intp(1)
75 for ax in axis:
---> 76 items *= arr.shape[mu.normalize_axis_index(ax, arr.ndim)]
77 else:
78 # TODO: Optimize case when `where` is broadcast along a non-reduction
AxisError: axis 2 is out of bounds for array of dimension 0
The target_layers is:
[BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)]
Can you please try looping over all the layers and setting requires grad before calling CAM?
for param in model.parameters():
param.requires_grad = False
If this works, then I think you probably want to return the model back to it's previous state (e.g if you are using this inside the training loop).
Then something like this could work in total:
Before calling the cam:
params_with_requires_grad = []
for param in model.parameters():
if param.requires_grad:
param.requires_grad = False
params_with_requires_grad.append(param)
Then reverting the model state back after calling cam:
for param in params_with_requires_grad:
param.requires_grad=True
Does this work ?
Calling:
for param in model.parameters():
param.requires_grad = False
before the cam doesn't work, but when I set param.requires_grad = True
it works.
So if I understand this correctly thee actual state of the model is more important than how I trained the model.
Thanks a lot @jacobgil! Have a nice Sunday
I' currently facing a strange behaviour when using GradCam for my tuned model. When I Fine-tune it from scratch everything works fine, but when I do Feature Extraction I get the mentioned error.
My Script for the training:
The GradCam Script which works for Fine-tuned models:
Any ideas where the problem could be?