ascendhuawei
commented
1 year ago Hi, Batch model inference is kind of 'parallel processing'. So, you should be able to see benefit of using multiple-batch. As I tested, for this example, the inference time of batch 1, 2, 4, 8 are actually very close on Ascend910, i.e., increasing batch size does not increase the total model execution time.
But, the available benefit also depends on the SoC capability, as well as the software stack. and the benefit is only for the model inference part, not including pre or post processing.
So, I will suggest:
- double check your timer code, only timing the model inference part
- try latest CANN version and see any difference
- if still not difference, you could try deeper profiling the model execution (check our documentation). It could be possible that, the SoC you are using (I guess Ascend310) might be already fully utilized even running One batch. In that case, even increasing batch size, you might not see obvious performance benefit.
RRRRRayyyyy
第一种是固定batch输入转换的om模型，input_shape="images:12,3,640,640" or input_shape="images:8,3,640,640" ,
测试结果
依然是整体运行时间随着batch增大而增大。
第二种使用动态batch时间，模型转换设置参数input_shape="images:-1,3,640,640" --dynamic_batch_size="1,8,12"
测试时间和前面也是一样的。参考了文档和教程，请问我是哪里还存在问题呢？
我在参考\python\level2_simple_inference\1_classification\googlenet_imagenet_dynamic_batch样例进行动态batch输入测试时发现，假设batch为1，8，16，无论我选择哪个batch，forward的时间都是差不多平均15ms左右一张图（例如耗时=15ms*batch的输入图片数量），好像batch是串行处理并不是并行处理，处理采用的是_execute_with_dynamic_batch_size这个算子，请问动态batch输入是选择这个算子还是其他算子呢，batch处理是并行还是串行呢？
