[Bug] 使用tools/analysis_tools/benchmark.py测试推理速度

[lyk595]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version (master) or latest version (3.x).

Task

I have modified the scripts/configs, or I'm working on my own tasks/models/datasets.

Branch

3.x branch https://github.com/open-mmlab/mmdetection/tree/3.x

Environment

pip

Reproduces the problem - code sample

Copyright (c) OpenMMLab. All rights reserved.

import argparse import os

from mmengine import MMLogger from mmengine.config import Config, DictAction from mmengine.dist import init_dist from mmengine.utils import mkdir_or_exist

from mmdet.utils import register_all_modules from mmdet.utils.benchmark import (DataLoaderBenchmark, DatasetBenchmark, InferenceBenchmark)

def parse_args(): parser = argparse.ArgumentParser(description='MMDet benchmark') parser.add_argument('config', help='test config file path') parser.add_argument('checkpoint', help='checkpoint file') parser.add_argument( '--task', choices=['inference', 'dataloader', 'dataset'], default='dataloader', help='Which task do you want to go to benchmark') parser.add_argument( '--repeat-num', type=int, default=1, help='number of repeat times of measurement for averaging the results') parser.add_argument( '--max-iter', type=int, default=2000, help='num of max iter') parser.add_argument( '--log-interval', type=int, default=50, help='interval of logging') parser.add_argument( '--num-warmup', type=int, default=5, help='Number of warmup') parser.add_argument( '--fuse-conv-bn', action='store_true', help='Whether to fuse conv and bn, this will slightly increase' 'the inference speed') parser.add_argument( '--dataset-type', choices=['train', 'val', 'test'], default='test', help='Benchmark dataset type. only supports train, val and test') parser.add_argument( '--work-dir', help='the directory to save the file containing ' 'benchmark metrics') parser.add_argument( '--cfg-options', nargs='+', action=DictAction, help='override some settings in the used config, the key-value pair ' 'in xxx=yyy format will be merged into config file. If the value to ' 'be overwritten is a list, it should be like key="[a,b]" or key=a,b ' 'It also allows nested list/tuple values, e.g. key="[(a,b),(c,d)]" ' 'Note that the quotation marks are necessary and that no white space ' 'is allowed.') parser.add_argument( '--launcher', choices=['none', 'pytorch', 'slurm', 'mpi'], default='none', help='job launcher') parser.add_argument('--local_rank', type=int, default=0) args = parser.parse_args() if 'LOCAL_RANK' not in os.environ: os.environ['LOCAL_RANK'] = str(args.local_rank) return args

def inference_benchmark(args, cfg, distributed, logger): benchmark = InferenceBenchmark( cfg, args.checkpoint, distributed, args.fuse_conv_bn, args.max_iter, args.log_interval, args.num_warmup, logger=logger) return benchmark

def dataloader_benchmark(args, cfg, distributed, logger): benchmark = DataLoaderBenchmark( cfg, distributed, args.dataset_type, args.max_iter, args.log_interval, args.num_warmup, logger=logger) return benchmark

def dataset_benchmark(args, cfg, distributed, logger): benchmark = DatasetBenchmark( cfg, args.dataset_type, args.max_iter, args.log_interval, args.num_warmup, logger=logger) return benchmark

def main(): register_all_modules()

args = parse_args()
cfg = Config.fromfile(args.config)

if args.cfg_options is not None:
    cfg.merge_from_dict(args.cfg_options)

distributed = False
if args.launcher != 'none':
    init_dist(args.launcher, **cfg.get('env_cfg', {}).get('dist_cfg', {}))
    distributed = True

log_file = None
if args.work_dir:
    log_file = os.path.join(args.work_dir, 'benchmark.log')
    mkdir_or_exist(args.work_dir)

logger = MMLogger.get_instance(
    'mmdet', log_file=log_file, log_level='INFO')

benchmark = eval(f'{args.task}_benchmark')(args, cfg, distributed, logger)
benchmark.run(args.repeat_num)

if name == 'main': main()

Reproduces the problem - command or script

python tools/analysis_tools/benchmark.py configs/my_coco_config/my_coco_config.py checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth

Reproduces the problem - error message

02/22 09:29:17 - mmengine - INFO - before build: 02/22 09:29:17 - mmengine - INFO - (GB) mem_used: 1.47 | uss: 1.14 | pss: 1.14 | total_proc: 1 02/22 09:29:17 - mmengine - WARNING - The "dataset" registry in mmdet did not set import location. Fallback to call mmdet.utils.register_all_modules instead. 02/22 09:29:17 - mmengine - WARNING - The "transform" registry in mmdet did not set import location. Fallback to call mmdet.utils.register_all_modules instead. loading annotations into memory... Done (t=0.00s) creating index... index created! 02/22 09:29:17 - mmengine - WARNING - The "data sampler" registry in mmdet did not set import location. Fallback to call mmdet.utils.register_all_modules instead. 02/22 09:29:17 - mmengine - INFO - after build: 02/22 09:29:17 - mmengine - INFO - (GB) mem_used: 1.47 | uss: 1.14 | pss: 1.14 | total_proc: 1 02/22 09:29:18 - mmengine - INFO - ============== Done ================== Traceback (most recent call last): File "analysis_tools/benchmark.py", line 134, in main() File "analysis_tools/benchmark.py", line 130, in main benchmark.run(args.repeat_num) File "/home/lyk/lyk/mmdetection-3.x/mmdet/utils/benchmark.py", line 103, in run results = self.average_multiple_runs(results) File "/home/lyk/lyk/mmdetection-3.x/mmdet/utils/benchmark.py", line 391, in average_multiple_runs f'Overall fps: {fpslist[0]:.1f} batch/s, ' ZeroDivisionError: division by zero

Additional information

No response

[mm-assistant[bot]]

We recommend using English or English & Chinese for issues so that we could have broader discussion.

[zytx121]

Hi @lyk595, Please check if the number of images in your test set is less than 2000.

[lyk595]

The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午9:59 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Hi @lyk595, Please check if the number of images in your test set is less than 2000.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[TsingAchieve]

May be the verification set is test set in your config?

---原始邮件--- 发件人: @.> 发送时间: 2023年5月18日(周四) 上午10:03 收件人: @.>; 抄送: "Pinde @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午9:59 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Hi @lyk595, Please check if the number of images in your test set is less than 2000.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

[lyk595]

Yes, my verification set is the same as my test set. So how do I change the configuration?

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:07 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

May be the verification set is test set in your config?

---原始邮件--- 发件人: @.> 发送时间: 2023年5月18日(周四) 上午10:03 收件人: @.>; 抄送: "Pinde @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

李宇凯
@.***

 

------------------ 原始邮件 ------------------
发件人: "open-mmlab/mmdetection" @.>;
发送时间: 2023年5月18日(星期四) 上午9:59
@.>;
@.**@.>;
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Hi @lyk595, Please check if the number of images in your test set is less than 2000.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.>
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[TsingAchieve]

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.
2. Using mmdetection r2.28.2, this version of benchmark can run normally.

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:21 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Yes, my verification set is the same as my test set. So how do I change the configuration?

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:07 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

May be the verification set is test set in your config?

---原始邮件---
发件人: @.>
发送时间: 2023年5月18日(周四) 上午10:03
收件人: @.>;
抄送: "Pinde @.**@.>;
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

李宇凯
@.***

 

------------------ 原始邮件 ------------------
发件人: "open-mmlab/mmdetection" @.>;
发送时间: 2023年5月18日(星期四) 上午9:59
@.>;
@.**@.>;
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Hi @lyk595, Please check if the number of images in your test set is less than 2000.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.>
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.>
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

[lyk595]

Is the '2000' in here changed to the number of images in my verification set? parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:37 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.

2. Using mmdetection r2.28.2, this version of benchmark can run normally.

   ------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:21 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Yes, my verification set is the same as my test set. So how do I change the configuration?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午10:07
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   May be the verification set is test set in your config?

   ---原始邮件---
   发件人: @.>
   发送时间: 2023年5月18日(周四) 上午10:03
   收件人: @.>;
   抄送: "Pinde @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午9:59
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Hi @lyk595, Please check if the number of images in your test set is less than 2000.

   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[TsingAchieve]

Maybe you can give it a try.

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:44 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Is the '2000' in here changed to the number of images in my verification set? parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:37 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.

2. Using mmdetection r2.28.2, this version of benchmark can run normally.

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午10:21
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Yes, my verification set is the same as my test set. So how do I change the configuration?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午10:07
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   May be the verification set is test set in your config?

   ---原始邮件---
   发件人: @.>
   发送时间: 2023年5月18日(周四) 上午10:03
   收件人: @.>;
   抄送: "Pinde @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午9:59
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Hi @lyk595, Please check if the number of images in your test set is less than 2000.

   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.>
   — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

[lyk595]

Thanks， I will try.

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:50 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Maybe you can give it a try.

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:44 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Is the '2000' in here changed to the number of images in my verification set?
parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')

李宇凯
@.***

 

------------------ 原始邮件 ------------------
发件人: "open-mmlab/mmdetection" @.>;
发送时间: 2023年5月18日(星期四) 上午10:37
@.>;
@.**@.>;
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.

2. Using mmdetection r2.28.2, this version of benchmark can run normally.

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午10:21
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Yes, my verification set is the same as my test set. So how do I change the configuration?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午10:07
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   May be the verification set is test set in your config?

   ---原始邮件---
   发件人: @.>
   发送时间: 2023年5月18日(周四) 上午10:03
   收件人: @.>;
   抄送: "Pinde @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

   李宇凯
   @.***

    

   ------------------ 原始邮件 ------------------
   发件人: "open-mmlab/mmdetection" @.>;
   发送时间: 2023年5月18日(星期四) 上午9:59
   @.>;
   @.**@.>;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Hi @lyk595, Please check if the number of images in your test set is less than 2000.

   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.>
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.>
   — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[lyk595]

I made the modification according to what you said, and now the following error appears:

(mmdet3) D:\MachineLearning\mmlab\mmdetection-3.x\tools>python analysis_tools/benchmark.py Traceback (most recent call last):   File "analysis_tools/benchmark.py", line 11, in <module>     from mmdet.utils.benchmark import (DataLoaderBenchmark, DatasetBenchmark,   File "d:\machinelearning\mmlab\mmdetection-3.x\mmdet\utils\benchmark.py", line 48, in <module>     def print_process_memory(p: psutil.Process, AttributeError: 'NoneType' object has no attribute 'Process'

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 上午10:50 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Maybe you can give it a try.

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "open-mmlab/mmdetection" @.&gt;; 发送时间:&nbsp;2023年5月18日(星期四) 上午10:44 @.&gt;; @.**@.&gt;; 主题:&nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Is the '2000' in here changed to the number of images in my verification set?
parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')

李宇凯
@.***

&amp;nbsp;

------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
发件人: "open-mmlab/mmdetection" @.&amp;gt;;
发送时间:&amp;nbsp;2023年5月18日(星期四) 上午10:37
@.&amp;gt;;
@.**@.&amp;gt;;
主题:&amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.

2. Using mmdetection r2.28.2, this version of benchmark can run normally.

   ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
   发件人: "open-mmlab/mmdetection" @.&amp;amp;gt;;
   发送时间:&amp;amp;nbsp;2023年5月18日(星期四) 上午10:21
   @.&amp;amp;gt;;
   @.**@.&amp;amp;gt;;
   主题:&amp;amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Yes, my verification set is the same as my test set. So how do I change the configuration?

   李宇凯
   @.***

   &amp;amp;amp;nbsp;

   ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
   发件人: "open-mmlab/mmdetection" @.&amp;amp;amp;gt;;
   发送时间:&amp;amp;amp;nbsp;2023年5月18日(星期四) 上午10:07
   @.&amp;amp;amp;gt;;
   @.**@.&amp;amp;amp;gt;;
   主题:&amp;amp;amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   May be the verification set is test set in your config?

   ---原始邮件---
   发件人: @.&amp;amp;amp;amp;gt;
   发送时间: 2023年5月18日(周四) 上午10:03
   收件人: @.&amp;amp;amp;amp;gt;;
   抄送: "Pinde @.**@.&amp;amp;amp;amp;gt;;
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?

   李宇凯
   @.***

   &amp;amp;amp;amp;amp;nbsp;

   ------------------&amp;amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;amp;nbsp;------------------
   发件人: "open-mmlab/mmdetection" @.&amp;amp;amp;amp;amp;gt;;
   发送时间:&amp;amp;amp;amp;amp;nbsp;2023年5月18日(星期四) 上午9:59
   @.&amp;amp;amp;amp;amp;gt;;
   @.**@.&amp;amp;amp;amp;amp;gt;;
   主题:&amp;amp;amp;amp;amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

   Hi @lyk595, Please check if the number of images in your test set is less than 2000.

   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.&amp;amp;amp;amp;amp;gt;
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.&amp;amp;amp;amp;gt;
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.&amp;amp;amp;gt;
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: @.&amp;amp;gt;
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: @.&amp;gt;
   — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.&gt; — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[TsingAchieve]

If it doesn't work, you can try installing version 2.28.2. I have conducted benchmark testing on this version and it can run smoothly.

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 中午11:09 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

I made the modification according to what you said, and now the following error appears:

(mmdet3) D:\MachineLearning\mmlab\mmdetection-3.x\tools&gt;python analysis_tools/benchmark.py Traceback (most recent call last): &nbsp; File "analysis_tools/benchmark.py", line 11, in <module&gt; &nbsp; &nbsp; from mmdet.utils.benchmark import (DataLoaderBenchmark, DatasetBenchmark, &nbsp; File "d:\machinelearning\mmlab\mmdetection-3.x\mmdet\utils\benchmark.py", line 48, in <module&gt; &nbsp; &nbsp; def print_process_memory(p: psutil.Process, AttributeError: 'NoneType' object has no attribute 'Process'

李宇凯 @.***

&nbsp;

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "open-mmlab/mmdetection" @.&gt;; 发送时间:&nbsp;2023年5月18日(星期四) 上午10:50 @.&gt;; @.**@.&gt;; 主题:&nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Maybe you can give it a try.

------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
发件人: "open-mmlab/mmdetection" @.&amp;gt;;
发送时间:&amp;nbsp;2023年5月18日(星期四) 上午10:44
@.&amp;gt;;
@.**@.&amp;gt;;
主题:&amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Is the '2000' in here changed to the number of images in my verification set?
parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')

李宇凯
@.***

&amp;amp;nbsp;

------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
发件人: "open-mmlab/mmdetection" @.&amp;amp;gt;;
发送时间:&amp;amp;nbsp;2023年5月18日(星期四) 上午10:37
@.&amp;amp;gt;;
@.**@.&amp;amp;gt;;
主题:&amp;amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.    
2. Using mmdetection r2.28.2, this version of benchmark can run normally.    

------------------ 原始邮件 ------------------    
发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;    
发送时间: 2023年5月18日(星期四) 上午10:21    
***@***.***>;    
***@***.******@***.***>;    
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)    

 Yes, my verification set is the same as my test set. So how do I change the configuration?     

 李宇凯     
 ***@***.***     

       

 ------------------ 原始邮件 ------------------     
 发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;     
 发送时间: 2023年5月18日(星期四) 上午10:07     
 ***@***.***>;     
 ***@***.******@***.***>;     
 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)     

  May be the verification set is test set in your config?      

  ---原始邮件---      
  发件人: ***@***.***>      
  发送时间: 2023年5月18日(周四) 上午10:03      
  收件人: ***@***.***>;      
  抄送: "Pinde ***@***.******@***.***>;      
  主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)      

   The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?       

   李宇凯       
   ***@***.***       

           

   ------------------ 原始邮件 ------------------       
   发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;       
   发送时间: 2023年5月18日(星期四) 上午9:59       
   ***@***.***>;       
   ***@***.******@***.***>;       
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)       

   Hi @lyk595, Please check if the number of images in your test set is less than 2000.       

   —       
   Reply to this email directly, view it on GitHub, or unsubscribe.       
   You are receiving this because you were mentioned.Message ID: ***@***.***>       
  —      
  Reply to this email directly, view it on GitHub, or unsubscribe.      
  You are receiving this because you commented.Message ID: ***@***.***>      
 —     
 Reply to this email directly, view it on GitHub, or unsubscribe.     
 You are receiving this because you were mentioned.Message ID: ***@***.***>     
—    
Reply to this email directly, view it on GitHub, or unsubscribe.    
You are receiving this because you commented.Message ID: ***@***.***>    

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.&amp;amp;gt;
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.&amp;gt;
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.&gt; — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

[lyk595]

Thanks, I will use it in version 2.28.2. However, please fix this bug so that it will work in version 3x.

李宇凯 @.***

 

------------------ 原始邮件 ------------------ 发件人: "open-mmlab/mmdetection" @.>; 发送时间: 2023年5月18日(星期四) 中午11:14 @.>; @.**@.>; 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

If it doesn't work, you can try installing version 2.28.2. I have conducted benchmark testing on this version and it can run smoothly.

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "open-mmlab/mmdetection" @.&gt;; 发送时间:&nbsp;2023年5月18日(星期四) 中午11:09 @.&gt;; @.**@.&gt;; 主题:&nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

I made the modification according to what you said, and now the following error appears:

(mmdet3) D:\MachineLearning\mmlab\mmdetection-3.x\tools&amp;gt;python analysis_tools/benchmark.py
Traceback (most recent call last):
&amp;nbsp; File "analysis_tools/benchmark.py", line 11, in <module&amp;gt;
&amp;nbsp; &amp;nbsp; from mmdet.utils.benchmark import (DataLoaderBenchmark, DatasetBenchmark,
&amp;nbsp; File "d:\machinelearning\mmlab\mmdetection-3.x\mmdet\utils\benchmark.py", line 48, in <module&amp;gt;
&amp;nbsp; &amp;nbsp; def print_process_memory(p: psutil.Process,
AttributeError: 'NoneType' object has no attribute 'Process'

李宇凯
@.***

&amp;nbsp;

------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
发件人: "open-mmlab/mmdetection" @.&amp;gt;;
发送时间:&amp;nbsp;2023年5月18日(星期四) 上午10:50
@.&amp;gt;;
@.**@.&amp;gt;;
主题:&amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Maybe you can give it a try.

------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
发件人: "open-mmlab/mmdetection" @.&amp;amp;gt;;
发送时间:&amp;amp;nbsp;2023年5月18日(星期四) 上午10:44
@.&amp;amp;gt;;
@.**@.&amp;amp;gt;;
主题:&amp;amp;nbsp;Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)

Is the '2000' in here changed to the number of images in my verification set?    
parser.add_argument('--max-iter', type=int, default=2000, help='num of max iter')    

李宇凯    
***@***.***    

     

------------------ 原始邮件 ------------------    
发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;    
发送时间: 2023年5月18日(星期四) 上午10:37    
***@***.***>;    
***@***.******@***.***>;    
主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)    

 1. You can try expanding the number of images in the validation set to 2000, or changing the number of images in the benchmark to the number of images in your validation set.     
 2. Using mmdetection r2.28.2, this version of benchmark can run normally.     

 ------------------ 原始邮件 ------------------     
 发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;     
 发送时间: 2023年5月18日(星期四) 上午10:21     
 ***@***.***>;     
 ***@***.******@***.***>;     
 主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)     

  Yes, my verification set is the same as my test set. So how do I change the configuration?      

  李宇凯      
  ***@***.***      

         

  ------------------ 原始邮件 ------------------      
  发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;      
  发送时间: 2023年5月18日(星期四) 上午10:07      
  ***@***.***>;      
  ***@***.******@***.***>;      
  主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)      

   May be the verification set is test set in your config?       

   ---原始邮件---       
   发件人: ***@***.***>       
   发送时间: 2023年5月18日(周四) 上午10:03       
   收件人: ***@***.***>;       
   抄送: "Pinde ***@***.******@***.***>;       
   主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)       

    The number of pictures in my training set is 300, the number of pictures in my verification set is 100, and there is no test set. What does it matter?        

    李宇凯        
    ***@***.***        

             

    ------------------ 原始邮件 ------------------        
    发件人:                                                                                                                        "open-mmlab/mmdetection"                                                                                    ***@***.***>;        
    发送时间: 2023年5月18日(星期四) 上午9:59        
    ***@***.***>;        
    ***@***.******@***.***>;        
    主题: Re: [open-mmlab/mmdetection] [Bug] 使用tools/analysis_tools/benchmark.py测试推理速度 (Issue #9815)        

    Hi @lyk595, Please check if the number of images in your test set is less than 2000.        

    —        
    Reply to this email directly, view it on GitHub, or unsubscribe.        
    You are receiving this because you were mentioned.Message ID: ***@***.***>        
   —       
   Reply to this email directly, view it on GitHub, or unsubscribe.       
   You are receiving this because you commented.Message ID: ***@***.***>       
  —      
  Reply to this email directly, view it on GitHub, or unsubscribe.      
  You are receiving this because you were mentioned.Message ID: ***@***.***>      
 —     
 Reply to this email directly, view it on GitHub, or unsubscribe.     
 You are receiving this because you commented.Message ID: ***@***.***>     
—    
Reply to this email directly, view it on GitHub, or unsubscribe.    
You are receiving this because you were mentioned.Message ID: ***@***.***>    

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.&amp;amp;gt;
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.&amp;gt;
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.&gt; — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>

[lujzz]

You can use pip install psutil to solve it.

[notbug1024]

In mmdet3.x，this might work. The value of "--max-iter" in the benchmark.py file is set to "the number of test parts in your data set" divided by "the value of batch_size of test_dataloader/val_dataloader in the config file"

[xyb1314]

pip install psutil

Thanks ! How do you find the solution?

[thetashall]

It is also possible that the number of datasets cannot be evenly divided by the batch. I didn't actually try, I didn't delve into the reasons behind it. Yolo has been able to run successfully, with batch size=1. Fast RCNN was unable to run at the beginning, but after switching to a different dataset, it ran successfully. After successfully running, batch size=4 is displayed,