search

NiccoloCavagnero / PEM

PEM: Prototype-based Efficient MaskFormer for Image Segmentation
https://niccolocavagnero.github.io/PEM/
66 stars 4 forks source link

about FPS and PQ #5

Closed nhw649 closed 4 months ago

nhw649 commented 4 months ago

I use two A100 GPUs following default configuration(run configs/ade20k/panoptic-segmentation/pem_R50_bs32_160k.yaml), PQ is about 37.1, and the FPS was only around 30.

[05/12 17:16:13 d2.evaluation.panoptic_evaluation]: Panoptic Evaluation Results: │················· | | PQ | SQ | RQ | #categories | │················· |:------:|:------:|:------:|:------:|:-------------:| │················· | All | 37.136 | 79.202 | 44.679 | 150 | │················· | Things | 35.976 | 80.007 | 43.646 | 100 | │················· | Stuff | 39.457 | 77.594 | 46.745 | 50 | │················· [05/12 17:16:13 d2.engine.defaults]: Evaluation results for ade20k_panoptic_val in csv format: │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: Task: sem_seg │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: mIoU,fwIoU,mACC,pACC │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: 44.2019,69.3469,56.5671,80.6667 │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: Task: panoptic_seg │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: PQ,SQ,RQ,PQ_th,SQ_th,RQ_th,PQ_st,SQ_st,RQ_st │················· [05/12 17:16:13 d2.evaluation.testing]: copypaste: 37.1363,79.2022,44.6789,35.9757,80.0066,43.6459,39.4574,77.5935,46.7449

NiccoloCavagnero commented 4 months ago

ADE20K is a very unstable dataset in terms of results, still we never experienced such drop in performance and we suggest to run the training again.

With respect to the FPS, there is also a high variability depending on the specific hardware setup, on how the FPS are computed. Other factors may affect the computation, especially the resolution of the images which is variable in ADE20K.

nhw649 commented 4 months ago

ADE20K is a very unstable dataset in terms of results, still we never experienced such drop in performance and we suggest to run the training again.

With respect to the FPS, there is also a high variability depending on the specific hardware setup, on how the FPS are computed. Other factors may affect the computation, especially the resolution of the images which is variable in ADE20K.

Can you provide model weights?

NiccoloCavagnero commented 4 months ago

We are planning to provide the pretrained models for the different configurations, but we do not know yet when they will be released.

NiccoloCavagnero commented 4 months ago

I am closing the issue if there are no additional doubts.