search

DengPingFan / PraNet

PraNet: Parallel Reverse Attention Network for Polyp Segmentation, MICCAI 2020 (Oral). Code using Jittor Framework is available.
http://dpfan.net/PraNet
437 stars 120 forks source link

Some question about Pranet #22

Closed zihuanqiu closed 3 years ago

zihuanqiu commented 3 years ago

I used the Kavsir data and the matlab eval program provided by you to do several sets of comparison experiments, and I have some question about paper's experiment part.

The following experiments use adam bs=16 lr=1e-4 and train for 20 epoches, which is same as yours

  1. resnet34 + Unet: meanDic:0.890;meanIoU:0.831;wFm:0.874;Sm:0.900;meanEm:0.944;MAE:0.031;maxEm:0.947;maxDice:0.893;maxIoU:0.834;meanSen:0.909;maxSen:1.000;meanSpe:0.978;maxSpe:0.982.

  2. res2net50_26w_4s + Unet: meanDic:0.895;meanIoU:0.842;wFm:0.888;Sm:0.908;meanEm:0.950;MAE:0.027;maxEm:0.953;maxDice:0.898;maxIoU:0.845;meanSen:0.896;maxSen:1.000;meanSpe:0.975;maxSpe:0.979.

  3. resnet34 + PraNet: meanDic:0.886;meanIoU:0.821;wFm:0.866;Sm:0.895;meanEm:0.940;MAE:0.031;maxEm:0.942;maxDice:0.889;maxIoU:0.823;meanSen:0.912;maxSen:1.000;meanSpe:0.969;maxSpe:0.973.

  4. res2net50_26w_4s + PraNet: meanDic:0.909;meanIoU:0.855;wFm:0.898;Sm:0.915;meanEm:0.956;MAE:0.025;maxEm:0.959;maxDice:0.912;maxIoU:0.858;meanSen:0.923;maxSen:1.000;meanSpe:0.980;maxSpe:0.984.

The following experiments use adam bs=16 lr=1e-4 and train for 40 epoches

  1. resnet34 + Unet: meanDic:0.912;meanIoU:0.855;wFm:0.900;Sm:0.916;meanEm:0.955;MAE:0.027;maxEm:0.958;maxDice:0.914;maxIoU:0.857;meanSen:0.919;maxSen:1.000;meanSpe:0.984;maxSpe:0.988.

  2. res2net50_26w_4s + Unet: meanDic:0.902;meanIoU:0.847;wFm:0.889;Sm:0.911;meanEm:0.955;MAE:0.025;maxEm:0.958;maxDice:0.905;maxIoU:0.849;meanSen:0.911;maxSen:1.000;meanSpe:0.982;maxSpe:0.986.

  3. resnet34 + PraNet: meanDic:0.900;meanIoU:0.844;wFm:0.888;Sm:0.909;meanEm:0.947;MAE:0.026;maxEm:0.950;maxDice:0.903;maxIoU:0.847;meanSen:0.916;maxSen:1.000;meanSpe:0.969;maxSpe:0.973.

  4. res2net50_26w_4s + PraNet: meanDic:0.908;meanIoU:0.856;wFm:0.901;Sm:0.916;meanEm:0.956;MAE:0.024;maxEm:0.959;maxDice:0.911;maxIoU:0.859;meanSen:0.909;maxSen:1.000;meanSpe:0.986;maxSpe:0.990.

My question is: Have you compare to Unet with the same backbone as Pranet? I think the result your paper reported may be the vanilla Unet. For fair compare, I trained Unet with resnet34 and res2net50_26w_4s, the result shows Pranet with few train epoches may have 1% Dice improvement. However, when train with enough epoches, Unet-resnet34 achieve 0.912 which seems outperform Pranet? Compared to Unet, to what extent can Pranet's decoder improve performance, and is it really effective? Thanks for your jobs and wish your reply

DengPingFan commented 3 years ago

Hi, Thanks for your attention and detailed experimental results.

Q. Have you compare to Unet with the same backbone as Pranet? I think the result your paper reported maybe the vanilla Unet. A. Ours is the first open-source work. Thus, we didn't have enough time to try other backbones. Thus, we only compare pranet with the vanilla Unet. Our main goal in this paper is to provide a possible solution for polyp segmentation.

Q. I trained Unet with resnet34 and res2net50_26w_4s, the result shows Pranet with few train epoches may have 1% Dice improvement. However, when train with enough epoches, Unet-resnet34 achieve 0.912 which seems outperform Pranet? Compared to Unet, to what extent can Pranet's decoder improve performance, and is it really effective? A. Thanks for your experiments. Indeed, from your experimental results. Our pranet is more efficient under 20 epochs. The reason we use 20 epoch is to adopt the early stop training strategy for better generalization ability. Note that we need to achieve a good balance on other datasets rather than only one dataset. Exploring to what extent can pranet's decoder improve performance is an interesting idea and we can do it in our journal version. We appreciate it if you could provide more interesting ideas and pros and cons of pranet architecture.

Interestingly, based on our recent video polyp segmentation experiments, we found that adopting the mobilenetv3 backbone may achieve comparable or even better performance. In addition, due to limited training data, lightweight backbone (e.g., mobilenet, resnet34) maybe works better than resenet50 and other similar.

Best,

Deng-Ping

zihuanqiu commented 3 years ago

Due to the cv2.imwrite function default to save as int8, it simply save the picture as 0 or 1 rather than probability between 0 and 1. So the experiment result I mentioned above may not correct.Please forgive me for the trouble caused to you.