search

PaddlePaddle / InterpretDL

InterpretDL: Interpretation of Deep Learning Models，基于『飞桨』的模型可解释性算法库。
https://interpretdl.readthedocs.io
Apache License 2.0
242 stars 38 forks source link

请问grad cam对于单通道图片输入应该办呢 #71

Open wanghonen opened 3 months ago

wanghonen commented 3 months ago

我使用grad cam做语义分割的可解释性，我训练好的模型也是单通道训练的，我是用gradcam的时候也是单通道，会报下面这个错误，请问我该怎么修改呢

AssertionError Traceback (most recent call last) /tmp/ipykernel_2663/1576993860.py in 4 pixels = [(30, 30)] 5 # good layers for deeplabv3p: backbone.bb_3_2.relu, head.aspp, head.decoder.conv ----> 6 exp = gradcam.interpret(img_path, pixels, target_layer_name='backbone.norm4', resize_to=512, crop_to=512)

/tmp/ipykernel_2663/3928282195.py in interpret(self, inputs, pixels, labels, target_layer_name, gradient_of, resize_to, crop_to, visual, save_path) 217 vis_explanation = explanation_to_vis(imgs[i], 218 gradcam_explanation[i], --> 219 style='overlay_heatmap') 220 if visual: 221 show_vis_explanation(vis_explanation)

~/.data/webide/pip/lib/python3.7/site-packages/interpretdl/data_processor/visualizer.py in explanation_to_vis(batched_image, explanation, style) 51 assert len(batched_image.shape) == 3 52 ---> 53 assert len(explanation.shape) == 2, f"image shape {batched_image.shape} vs " \ 54 f"explanation {explanation.shape}" 55

AssertionError: image shape (512, 512, 3) vs explanation (512,)

holyseven commented 3 months ago

image 试试把L103注释掉呢？https://github.com/PaddlePaddle/InterpretDL/blob/master/interpretdl/interpreter/gradient_cam.py#L103

wanghonen commented 3 months ago

image 试试把L103注释掉呢？https://github.com/PaddlePaddle/InterpretDL/blob/master/interpretdl/interpreter/gradient_cam.py#L103

十分感谢您的回复

1.我把这个注释之后确实不报错了，但是结果应该不对，显示的结果是纯的淡紫色，根本看不出东西。其实在报这个错误之前还报过一个错误，然后我把cam_weights = np.mean(g, (2, 3), keepdims=True)改成了cam_weights = np.mean(g, axis=2, keepdims=True)，我看了一下是因为返回的梯度是一个三维的，不是四维的，deeplabv3+返回的就是四维的，我的网络是segformer。 2.我用deeplabv3p试了一下，感觉效果也不好，虽然说有热图的样子，但是我测试了几个图，输出结果都是一样的，每太有区别 3.然后我又试了一下InputGrad_Segmentation，结果报这个错，你们有没有人能帮帮我，给点指导啊，崩溃了要

OSError Traceback (most recent call last) /tmp/ipykernel_57869/2151664618.py in 2 3 trained_model_path = 'https://bj.bcebos.com/paddleseg/dygraph/pascal_voc12/deeplabv3p_resnet50_os8_voc12aug_512x512_40k/model.pdparams' ----> 4 backbone = ResNet101_vd(multi_grid=(1,2,4)) 5 m = DeepLabV3P(num_classes=21, 6 backbone=backbone,

~/PaddleSeg/paddleseg/models/backbones/resnet_vd.py in ResNet101_vd(args) 387 @manager.BACKBONES.add_component 388 def ResNet101_vd(args): --> 389 model = ResNet_vd(layers=101, **args) 390 return model 391

~/PaddleSeg/paddleseg/models/backbones/resnet_vd.py in init(self, layers, output_stride, multi_grid, in_channels, pretrained, data_format) 259 stride=2, 260 act='relu', --> 261 data_format=data_format) 262 self.conv1_2 = ConvBNLayer( 263 in_channels=32,

~/PaddleSeg/paddleseg/models/backbones/resnet_vd.py in init(self, in_channels, out_channels, kernel_size, stride, dilation, groups, is_vd_mode, act, data_format) 59 groups=groups, 60 bias_attr=False, ---> 61 data_format=data_format) 62 63 self._batch_norm = layers.SyncBatchNorm(

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/conv.py in init(self, in_channels, out_channels, kernel_size, stride, padding, dilation, groups, padding_mode, weight_attr, bias_attr, data_format) 655 weight_attr=weight_attr, 656 bias_attr=bias_attr, --> 657 data_format=data_format) 658 659 def forward(self, x):

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/conv.py in init(self, in_channels, out_channels, kernel_size, transposed, dims, stride, padding, padding_mode, output_padding, dilation, groups, weight_attr, bias_attr, data_format) 134 shape=filter_shape, 135 attr=self._param_attr, --> 136 default_initializer=_get_default_param_initializer()) 137 self.bias = self.create_parameter( 138 attr=self._bias_attr, shape=[self._out_channels], is_bias=True)

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in create_parameter(self, shape, attr, dtype, is_bias, default_initializer) 422 temp_attr = None 423 return self._helper.create_parameter(temp_attr, shape, dtype, is_bias, --> 424 default_initializer) 425 426 @deprecated(

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layer_helper_base.py in create_parameter(self, attr, shape, dtype, is_bias, default_initializer, stop_gradient, type) 379 type=type, 380 stop_gradient=stop_gradient, --> 381 **attr._to_kwargs(with_initializer=True)) 382 else: 383 self.startup_program.global_block().create_parameter(

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py in create_parameter(self, *args, **kwargs) 3570 pass 3571 else: -> 3572 initializer(param, self) 3573 return param 3574

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/initializer.py in call(self, var, block) 365 out_var = _C_ops.gaussian_random( 366 'shape', var.shape, 'dtype', out_dtype, 'mean', self._mean, --> 367 'std', self._std_dev, 'seed', self._seed, 'use_mkldnn', False) 368 369 if var.dtype in [VarDesc.VarType.FP16, VarDesc.VarType.BF16]:

OSError: [operator < gaussian_random > error]

holyseven commented 3 months ago

  1. 看看你那边能复现这两个jupyter里的热图吗？https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/GradCAM_Segmentation.ipynb https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation.ipynb

  2. 我理解是单通道的图片效果不好，我这边没有单通道的模型，建议你先试试看，确认一下这几行结果的shape，调整一下适配单通道。

    image

  3. 我这边找个单通道模型试试

PS 另外基于patch作为token的transformer模型，不确定效果

wanghonen commented 3 months ago
  1. 看看你那边能复现这两个jupyter里的热图吗？https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/GradCAM_Segmentation.ipynb https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation.ipynb
  2. 我理解是单通道的图片效果不好，我这边没有单通道的模型，建议你先试试看，确认一下这几行结果的shape，调整一下适配单通道。
image
  1. 我这边找个单通道模型试试

PS 另外基于patch作为token的transformer模型，不确定效果

好的好的，我先试一下，如果你需要我这的模型和数据的话，我可以邮箱发给你，我的邮箱是985388035@qq.com, 十分感谢

wanghonen commented 3 months ago
  1. 看看你那边能复现这两个jupyter里的热图吗？https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/GradCAM_Segmentation.ipynb https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation.ipynb
  2. 我理解是单通道的图片效果不好，我这边没有单通道的模型，建议你先试试看，确认一下这几行结果的shape，调整一下适配单通道。
image
  1. 我这边找个单通道模型试试

PS 另外基于patch作为token的transformer模型，不确定效果

GradCAM是能复现的，效果看着也正常，inputgrad_segmentaton复现不了，会报我上面给的那个错误

holyseven commented 3 months ago

GradCAM是能复现的，效果看着也正常，inputgrad_segmentaton复现不了，会报我上面给的那个错误

这个我这边能正常跑，跑出的结果和链接中的区别不大。

我把deeplabv3改成了单通道的，然后跑了一下，可以看下这个：https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation_for_one_channel.ipynb

wanghonen commented 3 months ago

GradCAM是能复现的，效果看着也正常，inputgrad_segmentaton复现不了，会报我上面给的那个错误

这个我这边能正常跑，跑出的结果和链接中的区别不大。

我把deeplabv3改成了单通道的，然后跑了一下，可以看下这个：https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation_for_one_channel.ipynb 你好，非常感谢，我根据你给的链接，确实可以运行成功，不过目前关于inputgrad_segmentaton有几个问题

1.图片红框里面的两个模型路径可以都写自己训练好的模型吗 trained_model_path = '/home/aistudio/deeplabv3pmodel.pdparams' a = paddle.load("/home/aistudio/deeplabv3pmodel.pdparams") 微信图片编辑_20240822123551

2.我怎么知道可视化的是哪个层的呢，这个代码表示可视化的目标层吗，跟gradcam的代码不太一样， model_weights['backbone.conv1_1._conv.weight'] = a['backbone.conv1_1._conv.weight'][:, 2:3]

  1. 如果我想保存图片的话在哪里能修改代码呢，虽然安装了interpreedl，但是我在项目里面找不到相关安装文件，我用的是paddleseg aistudio
  2. 能麻烦你再弄个gradcam的单通道的demo吗，我按照你改的inputgrad_segmentaton的代码去修改，还是汇报通道不匹配的错误 ValueError: (InvalidArgument) The number of input's channels should be equal to filter's channels groups for Op(Conv). But received: the input's channels is 3, the input's shape is [1, 3, 512, 512]; the filter's channels is 1, the filter's shape is [32, 1, 3, 3]; the groups is 1, the data_format is NCHW. The error may come from wrong data_format setting. [Hint: Expected input_channels == filter_dims[1] groups, but received input_channels:3 != filter_dims[1] * groups:1.] (at /paddle/paddle/fluid/operators/conv_op.cc:116) [operator < conv2d > error] 5.不知道你能试试segformer的单通道模型效果吗，不知道是不是我这的问题，感觉效果不好
wanghonen commented 3 months ago

GradCAM是能复现的，效果看着也正常，inputgrad_segmentaton复现不了，会报我上面给的那个错误

这个我这边能正常跑，跑出的结果和链接中的区别不大。

我把deeplabv3改成了单通道的，然后跑了一下，可以看下这个：https://github.com/PaddlePaddle/InterpretDL/blob/master/tutorials/InputGrad_Segmentation_for_one_channel.ipynb

还有个问题是为什么segformer产生的gradients是三维[1, 256, 512]的，而deeplabv3+产生的gradients是四维[1, 2048, 64, 64]的呢

holyseven commented 3 months ago

1.图片红框里面的两个模型路径可以都写自己训练好的模型吗 2.我怎么知道可视化的是哪个层的呢，这个代码表示可视化的目标层吗，跟gradcam的代码不太一样，

可以的，注意：那个demo只是为了把3通道的模型转成单通道用的，相关代码都是为了适配单通道（包括模型部分和输入部分的代码修改）。

input_grad的解释算法默认是对输入层做的，对transformer的模型，需要调成对embedding层，但是效果需要调，不一定好，对segmentation没做过，需要自己尝试一下。

  1. 如果我想保存图片的话在哪里能修改代码呢
image

在这里修改

  1. 能麻烦你再弄个gradcam的单通道的demo吗

原理都是一样的，把模型和输入改了，其它都是一样的。

PS 另外基于patch作为token的transformer模型，不确定效果

还有个问题是为什么segformer产生的gradients是三维[1, 256, 512]的，而deeplabv3+产生的gradients是四维[1, 2048, 64, 64]的呢

demo里用的deeplabv3p，gradients也是[512,512]。