search

mcordts / cityscapesScripts

README and scripts for the Cityscapes Dataset
MIT License
2.19k stars 608 forks source link

fix the precision value for the first point on the pr-rc curve #171

Closed abred closed 7 months ago

abred commented 1 year ago

Setting the precision value for recall=0 to 1 can lead to significant overestimation. The fewer points are on the curve the higher the impact. For typical proposal-based setups this is often not significant, however for proposal-free setups (which basically results in the curve having only a single point) this is significant. In one of our experiments (real model trained on real data) this lead to an overestimation of 6 percentage points.

To fix it the precision at recall=0 should be set to the same precision as at the lowest recall value.

mcordts commented 7 months ago

I'll reject the PR as I would like these scripts and the reported metrics to be consistent with our evaluation server.