search

dvlab-research / PanopticFCN

Fully Convolutional Networks for Panoptic Segmentation (CVPR2021 Oral)
Apache License 2.0
391 stars 53 forks source link

Question about kernel_fusion in head.py #7

Closed lucasjinreal closed 3 years ago

lucasjinreal commented 3 years ago

Hi, I found that kernel_fusion only called in inference step in ThingsGenerator. May I ask what's this purpose and why only do that in inference?

lucasjinreal commented 3 years ago

Actually I tried uncomment it when inference, seems not effect the results.

yanwei-li commented 3 years ago

Thanks for these questions. The response is given as follows:

  1. For the reason that each point within the object is utilized to represent the whole instance in the training stage (consider k=1 in Tab. 6), the kernel_fusion (aims to merge the kernels of the same identity) can be omitted for concision. And we also tried to use kernel_fusion to merge all the kernels (when k>1 in Tab. 6, this means k points are utilized together to represent one instance), but found much inferior performance. This could be attributed to that it could not perfectly match the training process (k kernels belong to the same instance are needed) in the inference stage, which is also in conflict with our prior (each point is utilized to represent the whole instance).
  2. Uncomment the kernel_fusion actually will bring the performance drop, which is also reflected in the experiment of Tab. 3. Maybe you could attach the detailed code for reference.
lucasjinreal commented 3 years ago

@yanwei-li Thank u for your reply. I notice that if comment out will bring overlapped boxes but over all result seems normal. the kernel fusion step can be replaced by some common ops in pytorch? I am current want converting model inference steps to tensorrt, seems postprocessing a little challenging considering this step as well

yanwei-li commented 3 years ago

I attempt to comment out the kernel_fusion and listed the results of PanopticFCN-R50-3x as follows:

Method    | PQ  |  SQ |  RQ |  AP |  mIoU
----------------------------------------
no fusion | 42.8 | 81.1 | 51.6 | 31.3 | 42.0
fusion    | 43.6 | 81.4 | 52.5 | 34.4 | 43.6

Compared with the original results with kernel_fusion, it damages the performance a lot, specifically in AP and mIoU. I have not tried some other replacements for kernel fusion. Maybe some operations like AveragePooling could work if it is utilized in an appropriate method.

lucasjinreal commented 3 years ago

@yanwei-li thank u for your suggestions!