Lightning-AI / torchmetrics

Machine learning metrics for distributed, scalable PyTorch applications.
https://lightning.ai/docs/torchmetrics/
Apache License 2.0
2.11k stars 403 forks source link

Torchetrics MeanAveragePrecision compability with numpy >1.20.0 #2035

Closed Stasolet closed 1 year ago

Stasolet commented 1 year ago

🐛 Bug

To Reproduce

Code sample used from documentation: ```python from torch import tensor from torchmetrics.detection import MeanAveragePrecision preds = [ dict( boxes=tensor([[258.0, 41.0, 606.0, 285.0]]), scores=tensor([0.536]), labels=tensor([0]), ) ] target = [ dict( boxes=tensor([[214.0, 41.0, 562.0, 285.0]]), labels=tensor([0]), ) ] metric = MeanAveragePrecision(iou_type="bbox") metric.update(preds, target) from pprint import pprint pprint(metric.compute()) ```

Environment

Additional context

In documentation for map I found "This metric utilizes the official pycocotools implementation as its backend". But there is a small confusing point: the official repository "pycocotools" and "pycocotools" package in pypi are different things. Which official pycocotools package was meant in the documentation?

github-actions[bot] commented 1 year ago

Hi! thanks for your contribution!, great first issue!

SkafteNicki commented 1 year ago

Hi @Stasolet, thanks for reporting this issue.

Regarding the first issue of compability with numpy >1.20.0: There is really not mush we can do here as long as it does not get fixed in the official repository. I know there are open PRs to fix it, but since the repository is not really maintained anymore I do really not have high hopes. I recommend install the fork on pypi as that is maintained and supports numpy > 1.20.0.

Regarding the confusion about the documentation: You are completely right that the language in the note is not explicit enough. Essentially, we only detect if a package named pycocotools is installed, not where it is installed from. Therefore, we support both the official implementation and fork found on pypi because both installs as pycocotools. But this is not clear. I fixed the note in PR https://github.com/Lightning-AI/torchmetrics/pull/2034 to make it more clear what we actually mean where we also add support for another backend: https://github.com/MiXaiLL76/faster_coco_eval