edwardzhou130
commented
3 years ago Sorry, I did not use a random seed when I did my experiment. All parameters should be the same. The only difference might be in: https://github.com/edwardzhou130/Panoptic-PolarNet/blob/3a72f2380a4e505e191b69da596f521a9d9f1a71/dataloader/instance_augmentation.py#L163 There was a bug before. But the only difference is whether the loaded instance will be rotated along the center or not.
Thank you for your work! I ran two experiments to reproduce the results of the pretrained model listed in repository:
Results for the pretrained model:
Validation per class PQ, SQ, RQ and IoU:
car : 87.94% 91.18% 96.45% 93.65%
bicycle : 52.57% 77.49% 67.84% 46.37%
motorcycle : 65.87% 88.72% 74.24% 66.71%
truck : 62.52% 87.94% 71.10% 88.46%
bus : 52.99% 88.41% 59.93% 43.35%
person : 73.23% 89.91% 81.45% 62.23%
bicyclist : 85.31% 90.66% 94.11% 82.21%
motorcyclist : 41.43% 84.35% 49.12% 48.48%
road : 93.74% 93.74% 100.00% 93.63%
parking : 24.17% 70.29% 34.38% 45.05%
sidewalk : 76.44% 83.27% 91.80% 80.26%
other-ground : 0.00% 0.00% 0.00% 0.48%
building : 87.46% 90.54% 96.60% 90.28%
fence : 18.05% 70.02% 25.77% 51.68%
vegetation : 84.88% 85.51% 99.26% 86.44%
trunk : 45.25% 72.43% 62.47% 61.96%
terrain : 56.98% 74.62% 76.36% 73.54%
pole : 55.89% 73.23% 76.32% 62.35%
traffic-sign : 54.20% 73.82% 73.43% 46.23%
Current val PQ is 58.890
Current val miou is 64.387
Results of my experiments:
Validation per class PQ, SQ, RQ and IoU:
car : 85.57% 89.42% 95.70% 92.69%
bicycle : 45.07% 77.62% 58.06% 38.38%
motorcycle : 60.85% 87.75% 69.34% 53.97%
truck : 66.72% 89.95% 74.17% 73.40%
bus : 52.96% 88.94% 59.55% 35.49%
person : 69.85% 89.19% 78.32% 56.55%
bicyclist : 87.17% 91.88% 94.87% 77.88%
motorcyclist : 41.59% 80.88% 51.43% 50.53%
road : 92.43% 92.45% 99.98% 92.30%
parking : 22.38% 68.82% 32.52% 39.85%
sidewalk : 74.24% 80.88% 91.80% 77.54%
other-ground : 0.00% 0.00% 0.00% 0.30%
building : 86.14% 90.27% 95.43% 88.60%
fence : 17.57% 67.88% 25.89% 48.37%
vegetation : 83.11% 85.61% 97.08% 85.21%
trunk : 35.16% 70.44% 49.91% 53.74%
terrain : 54.49% 73.21% 74.43% 69.84%
pole : 55.85% 73.17% 76.33% 62.68%
traffic-sign : 52.02% 75.09% 69.28% 43.30%
Current val PQ is 57.010
Current val miou is 60.034
car : 87.62% 91.00% 96.29% 93.89%
bicycle : 45.15% 77.46% 58.28% 42.56% motorcycle : 64.76% 89.65% 72.24% 64.91% truck : 54.76% 86.15% 63.56% 59.50% bus : 47.48% 89.10% 53.29% 37.47% person : 68.85% 89.77% 76.69% 54.36% bicyclist : 84.63% 89.93% 94.11% 82.65% motorcyclist : 32.75% 88.76% 36.89% 38.15% road : 93.18% 93.18% 100.00% 93.19% parking : 18.76% 66.70% 28.13% 40.71% sidewalk : 76.08% 81.98% 92.80% 79.75% other-ground : 0.00% 0.00% 0.00% 0.00% building : 86.66% 89.91% 96.39% 89.58% fence : 16.05% 67.93% 23.63% 49.75% vegetation : 82.70% 85.32% 96.93% 85.06% trunk : 42.30% 71.85% 58.88% 58.86% terrain : 53.89% 73.51% 73.31% 70.57% pole : 55.80% 72.88% 76.56% 62.47% traffic-sign : 51.75% 74.62% 69.36% 42.06%
Current val PQ is 55.956 Current val miou is 60.289
Both experiments were running for almost 100 epochs and their results (mIoU 60.3) don't match the results of your pretrained model (mIoU 64.3)
Could you please share the parameters and random seed for the experiment that achieved mIoU 64.3?
Thank you!