Regarding evaluation results.

[deepAICrazy]

Hi authors, Thanks for sharing your code. In your code, you resize the images to [400,400] to evaluate both PASCAL5i and COCO20i. PFENet proposes to evaluate with the original label and REPRI use 417 as the evaluation size, but in their codes, the aspect ratios of images is the same as the original images. As
https://github.com/juhongm999/hsnet/issues/1#issuecomment-816485819 and the results in the paper of PFENet, resizing labels to smaller sizes will bring significant improvement because details are missing in lower resolutions and details are difficult. [400,400] is even much smaller than [473,473], so additional improvement will be made by the small evaluation size [400,400]. A recent cvpr21 accepted paper https://arxiv.org/pdf/2104.01893.pdf also uses the above evaluation strategy for a fair comparison So I think you should also evaluate your models with the original labels, or make their aspect ratios are kept as the original ones like ASGNet in [473,473] to made a fair comparison with recent papers.

[juhongm999]

Hello,

• PFENet proposes to evaluate with the original label and REPRI use 417 as the evaluation size, but in their codes, the aspect ratios of images is the same as the original images.

Answer: the code uses the same aspect ratios of the images but in the paper, they report results of both original aspect ratios and 1:1 aspect ratio. Please note that we compare our results with PFENet results of 1:1 aspect ratio for fair comparison.

• resizing labels to smaller sizes will bring significant improvement because details are missing in lower resolutions and details are difficult. [400,400] is even much smaller than [473,473], so additional improvement will be made by the small evaluation size [400,400].

Answer: as mentioned in issue #1, In our experiments, we found that larger image sizes typically (> 417) result in better mIoU results but we use image size of 400 x 400 as the image sizes of 417x417 and 473x473 seemed random numbers to us and our model with the size of 400 x 400 already surpassed previous state of the arts, meaning that we actually got better results with larger image sizes (> 400) in our experiment. However, given larger computational complexity induced by larger images, we used smaller image size of 400x400. You can try this on your own as the code is provided.

[deepAICrazy]

This is not the case actually because you did not fairly compare your results with REPRI and PFENet in the table1 where the results are directly copied from their papers. In https://github.com/mboudiaf/RePRI-for-Few-Shot-Segmentation/blob/master/src/dataset/transform.py#L80 they keep the aspect ratio of the resized images to be the same as the original image, but in your implementation https://github.com/juhongm999/hsnet/blob/e288916debe5290b3e9554fb61e13a474e00f885/data/dataset.py#L25 the images are simply resized to be the aspect ratio 1:1 without keeping the original label.

For my second question, now that all previous methods use 417, 473 or the original sizes for evaluating COCO and PASCAL. I do not understand why did you use size 400 on COCO and PASCAL to create a brand **new** setting and make other people hard to follow to have a fair comparison, even if the size 400 does not bring the best performance according to your words. Normally we should show the setting of the best results. This is true the performance on Pascal will be slightly higher when the training size grows but they are still comparable. And the models of REPRI and PASCAL cannot be directly tested with 1:1 aspect ratio because they are not trained with the images the 1:1 ratios in the non-255 regions. However on COCO I have tested on PFENet. the results will be much lower when it is evaluated with the original labels without resize. it is the same to the results shownin PFENet it is also mentioned by https://github.com/juhongm999/hsnet/issues/1#issuecomment-816485819. So I think it is unfair if you cannot show the COCO results with the original aspects and the original sizes (or 417, 473) to compare with related methods (REPRI, PFENET AND ASGNet and so on) because smaller size does bring much better performance on COCO.