[Bug] The model can't get the eval_map (mAp) output.

BambaSpoid commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I have modified the scripts/configs, or I'm working on my own tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

Here is the environment I use:

Pytorch version : 1.12.1+cu116
Torchvision : 0.13.1
Torchaudio :0.12.1
Torchtext : 0.13.1
Mmcv-full : 1.6.2

I run this code for the installation

!pip3 install -U openmim
!mim install mmcv-full=='1.6.2' -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12/index.html
!mim install mmdet
!pip install mmengine

And here is the result

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting openmim
  Downloading openmim-0.3.6-py2.py3-none-any.whl (51 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.3/51.3 KB 5.9 MB/s eta 0:00:00
Collecting model-index
  Downloading model_index-0.1.11-py3-none-any.whl (34 kB)
Collecting rich
  Downloading rich-13.3.1-py3-none-any.whl (239 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.0/239.0 KB 25.7 MB/s eta 0:00:00
Collecting colorama
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from openmim) (1.3.5)
Requirement already satisfied: Click in /usr/local/lib/python3.8/dist-packages (from openmim) (8.1.3)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from openmim) (2.25.1)
Requirement already satisfied: tabulate in /usr/local/lib/python3.8/dist-packages (from openmim) (0.8.10)
Requirement already satisfied: pip>=19.3 in /usr/local/lib/python3.8/dist-packages (from openmim) (22.0.4)
Collecting ordered-set
  Downloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)
Requirement already satisfied: markdown in /usr/local/lib/python3.8/dist-packages (from model-index->openmim) (3.4.1)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from model-index->openmim) (6.0)
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.8/dist-packages (from pandas->openmim) (1.22.4)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->openmim) (2022.7.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->openmim) (2.8.2)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->openmim) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->openmim) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->openmim) (1.26.14)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->openmim) (2022.12.7)
Requirement already satisfied: typing-extensions<5.0,>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from rich->openmim) (4.5.0)
Collecting pygments<3.0.0,>=2.14.0
  Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 56.6 MB/s eta 0:00:00
Collecting markdown-it-py<3.0.0,>=2.1.0
  Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 KB 9.1 MB/s eta 0:00:00
Collecting mdurl~=0.1
  Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas->openmim) (1.15.0)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.8/dist-packages (from markdown->model-index->openmim) (6.0.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.4->markdown->model-index->openmim) (3.15.0)
Installing collected packages: pygments, ordered-set, mdurl, colorama, markdown-it-py, rich, model-index, openmim
  Attempting uninstall: pygments
    Found existing installation: Pygments 2.6.1
    Uninstalling Pygments-2.6.1:
      Successfully uninstalled Pygments-2.6.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.9.0 requires jedi>=0.10, which is not installed.
Successfully installed colorama-0.4.6 markdown-it-py-2.2.0 mdurl-0.1.2 model-index-0.1.11 openmim-0.3.6 ordered-set-4.1.0 pygments-2.14.0 rich-13.3.1
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Looking in links: https://download.openmmlab.com/mmcv/dist/cu116/torch1.12/index.html, https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html
Collecting mmcv-full==1.6.2
  Downloading https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/mmcv_full-1.6.2-cp38-cp38-manylinux1_x86_64.whl (40.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.2/40.2 MB 11.9 MB/s eta 0:00:00
Requirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from mmcv-full==1.6.2) (6.0)
Collecting yapf
  Downloading yapf-0.32.0-py2.py3-none-any.whl (190 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.2/190.2 KB 12.8 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from mmcv-full==1.6.2) (23.0)
Requirement already satisfied: Pillow in /usr/local/lib/python3.8/dist-packages (from mmcv-full==1.6.2) (8.4.0)
Collecting addict
  Downloading addict-2.4.0-py3-none-any.whl (3.8 kB)
Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.8/dist-packages (from mmcv-full==1.6.2) (4.6.0.66)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from mmcv-full==1.6.2) (1.22.4)
Installing collected packages: yapf, addict, mmcv-full
Successfully installed addict-2.4.0 mmcv-full-1.6.2 yapf-0.32.0
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Looking in links: https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html
Collecting mmdet
  Downloading mmdet-2.28.2-py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 59.3 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from mmdet) (1.7.3)
Requirement already satisfied: pycocotools in /usr/local/lib/python3.8/dist-packages (from mmdet) (2.0.6)
Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from mmdet) (1.15.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from mmdet) (1.22.4)
Collecting terminaltables
  Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from mmdet) (3.5.3)
Requirement already satisfied: mmcv-full>=1.3.17 in /usr/local/lib/python3.8/dist-packages (from mmdet) (1.6.2)
Requirement already satisfied: Pillow in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (8.4.0)
Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (4.6.0.66)
Requirement already satisfied: addict in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (2.4.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (23.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (6.0)
Requirement already satisfied: yapf in /usr/local/lib/python3.8/dist-packages (from mmcv-full>=1.3.17->mmdet) (0.32.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmdet) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmdet) (1.4.4)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmdet) (0.11.0)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmdet) (3.0.9)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmdet) (4.38.0)
Installing collected packages: terminaltables, mmdet
Successfully installed mmdet-2.28.2 terminaltables-3.1.10
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting mmengine
  Downloading mmengine-0.6.0-py3-none-any.whl (360 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 360.7/360.7 KB 21.7 MB/s eta 0:00:00
Requirement already satisfied: termcolor in /usr/local/lib/python3.8/dist-packages (from mmengine) (2.2.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from mmengine) (6.0)
Requirement already satisfied: yapf in /usr/local/lib/python3.8/dist-packages (from mmengine) (0.32.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from mmengine) (3.5.3)
Requirement already satisfied: rich in /usr/local/lib/python3.8/dist-packages (from mmengine) (13.3.1)
Requirement already satisfied: addict in /usr/local/lib/python3.8/dist-packages (from mmengine) (2.4.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from mmengine) (1.22.4)
Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.8/dist-packages (from mmengine) (4.6.0.66)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (1.4.4)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (8.4.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (0.11.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (2.8.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (23.0)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (3.0.9)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->mmengine) (4.38.0)
Requirement already satisfied: markdown-it-py<3.0.0,>=2.1.0 in /usr/local/lib/python3.8/dist-packages (from rich->mmengine) (2.2.0)
Requirement already satisfied: typing-extensions<5.0,>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from rich->mmengine) (4.5.0)
Requirement already satisfied: pygments<3.0.0,>=2.14.0 in /usr/local/lib/python3.8/dist-packages (from rich->mmengine) (2.14.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.8/dist-packages (from markdown-it-py<3.0.0,>=2.1.0->rich->mmengine) (0.1.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7->matplotlib->mmengine) (1.15.0)
Installing collected packages: mmengine
Successfully installed mmengine-0.6.0

Reproduces the problem - code sample

from mmdet.apis import set_random_seed

# Modify dataset type and path
cfg.dataset_type = 'PostDataset'
cfg.data_root = '/content/ammi-research-project/mmrotate/datasets'

cfg.data.test.type = 'PostDataset'
cfg.data.test.data_root = '/content/ammi-research-project/mmrotate/datasets/test'
cfg.data.test.ann_file = 'labelTxt'
cfg.data.test.img_prefix = 'images'

cfg.data.train.type = 'PostDataset'
cfg.data.train.data_root = '/content/ammi-research-project/mmrotate/datasets/train'
cfg.data.train.ann_file = 'labelTxt'
cfg.data.train.img_prefix = 'images'

cfg.data.val.type = 'PostDataset'
cfg.data.val.data_root = '/content/ammi-research-project/mmrotate/datasets/valid'
cfg.data.val.ann_file = 'labelTxt'
cfg.data.val.img_prefix = 'images'

# modify num classes of the model in box head
cfg.model.bbox_head.num_classes = 7
# We can still use the pre-trained Mask RCNN model though we do not need to
# use the mask branch
cfg.load_from = 'rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth'

# Set up working dir to save files and logs.
cfg.work_dir = './tutorial_exps'

cfg.optimizer.lr = 0.001
cfg.lr_config.warmup = None
cfg.runner.max_epochs = 1
cfg.log_config.interval = 10

# Change the evaluation metric since we use customized dataset.
cfg.evaluation.metric = 'mAP'
# We can set the evaluation interval to reduce the evaluation times
cfg.evaluation.interval = 1
# We can set the checkpoint saving interval to reduce the storage cost
cfg.checkpoint_config.interval = 1

# Set seed thus the results are more reproducible
cfg.seed = 0
set_random_seed(0, deterministic=False)
cfg.gpu_ids = range(1)
cfg.device='cuda'

# We can also use tensorboard to log the training process
cfg.log_config.hooks = [
    dict(type='TextLoggerHook'),
    dict(type='TensorboardLoggerHook')]

# We can initialize the logger for training and have a look
# at the final config used for training
print(f'Config:\n{cfg.pretty_text}')

Reproduces the problem - command or script

import os.path as osp

from mmdet.datasets import build_dataset
from mmdet.models import build_detector
from mmdet.apis import train_detector

# Build dataset
datasets = [build_dataset(cfg.data.train)]
# print(datasets[0])

# Build the detector
model = build_detector(
    cfg.model, train_cfg=cfg.get('train_cfg'), test_cfg=cfg.get('train_cfg'))
# Add an attribute for visualization convenience  
model.CLASSES = datasets[0].CLASSES

# Create work_dir
mmcv.mkdir_or_exist(osp.abspath(cfg.work_dir))
train_detector(model, datasets, cfg, distributed=False, validate=True)

Reproduces the problem - error message

2023-03-02 11:47:44,157 - mmdet - INFO - Epoch [1][310/318] lr: 1.000e-03, eta: 0:00:04, time: 0.565, data_time: 0.012, memory: 3607, loss_cls: 0.8306, loss_bbox: 1.1637, loss: 1.9942, grad_norm: 7.7591
2023-03-02 11:47:48,846 - mmdet - INFO - Saving checkpoint at 1 epochs
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 34/34, 2.9 task/s, elapsed: 12s, ETA:     0s

I have waiting for more than hours without result (for just epoch), it just keeps turning

Additional information

I expect to see this table result below:

+-------------+-----+------+--------+-------+
| class       | gts | dets | recall | ap    |
+-------------+-----+------+--------+-------+
| cable       | 175 | 1083 | 0.211  | 0.032 |
| harrow      | 20  | 71   | 0.100  | 0.018 |
| iacm        | 4   | 7    | 0.000  | 0.000 |
| insulator   | 42  | 241  | 0.524  | 0.334 |
| lightening  | 16  | 105  | 0.438  | 0.271 |
| post        | 48  | 355  | 0.729  | 0.633 |
| transformer | 4   | 13   | 0.000  | 0.000 |
+-------------+-----+------+--------+-------+
| mAP         |     |      |        | 0.184 |
+-------------+-----+------+--------+-------+

zytx121 commented 1 year ago

Hi @BambaSpoid, please try to use https://github.com/open-mmlab/mmrotate/tree/1.x. Due to the staff shortage, the master branch has stopped maintenance.

BambaSpoid commented 1 year ago

Hi @zytx121, thank you for returning me, I tested it but it returns a bug

need at least one array to concatenate

BambaSpoid commented 1 year ago

Hi @zytx121, thank you for returning me, I tested it but it returns a bug
need at least one array to concatenate

👆🏾Problem fixed, But still don't have solution for the first issue

SikGek commented 1 year ago

Were you able to find a solution to this? I'm currently on the main branch and this is happening with the exact same code.

BambaSpoid commented 1 year ago

Were you able to find a solution to this? I'm currently on the main branch and this is happening with the exact same code.

Hi @SikGek, You have to check your configuration, Choose the environment below, Python version : 3.8.10 or 3.8.15 Pytorch version : 1.12.1+cu116 Torchvision : 0.13.1 Torchaudio :0.12.1 Torchtext : 0.13.1 Mmcv-full : 1.6.2

open-mmlab / mmrotate