ValueError: need at least one array to concatenate

[dingtom]

使用pycharm进行代码修改并sftp推到服务器上训练遇到问题。windows系统执行sh文件进行训练没有问题但是使用Ubuntu系统执行sh文件报错，但是在pycharm中使用远程解释器进行训练又可以训练没报错，下面是sh文件内容和报错信息。

python tools/train.py \
configs/my/ssd300_coco.py \
--work-dir runs            \
--resume                    \
--amp
#--resume                    \

------------------------------------------------------------
System environment:
    sys.platform: linux
    Python: 3.9.0 (default, Nov 15 2020, 14:28:56) [GCC 7.3.0]
    CUDA available: True
    numpy_random_seed: 746404957
    GPU 0: NVIDIA GeForce RTX 2070 SUPER
    CUDA_HOME: /usr/local/cuda
    NVCC: Cuda compilation tools, release 11.7, V11.7.64
    GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
    PyTorch: 2.0.0+cu117
    PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.7
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.9  (built against CUDA 11.8)
    - Built with CuDNN 8.5
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

    TorchVision: 0.15.1+cu117
    OpenCV: 4.7.0
    MMEngine: 0.8.4

Runtime environment:
    cudnn_benchmark: False
    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
    dist_cfg: {'backend': 'nccl'}
    seed: 746404957
    Distributed launcher: none
    Distributed training: False
    GPU number: 1
------------------------------------------------------------

09/12 15:44:09 - mmengine - INFO - Config:
METAINFO = dict(
    classes=(
        'bulk cargo carrier',
        'ore carrier',
        'fishing boat',
        'general cargo ship',
        'container ship',
        'passenger ship',
    ),
    palette=[
        (
            220,
            20,
            60,
        ),
        (
            119,
            11,
            32,
        ),
        (
            0,
            0,
            142,
        ),
        (
            0,
            0,
            230,
        ),
        (
            106,
            0,
            228,
        ),
        (
            0,
            60,
            100,
        ),
    ])
auto_scale_lr = dict(base_batch_size=16, enable=False)
backend_args = None
cudnn_benchmark = True
custom_hooks = [
    dict(type='NumClassCheckHook'),
    dict(interval=50, priority='VERY_LOW', type='CheckInvalidLossHook'),
]
data_root = '/srv/samba/dingwenchao/SeaShips/COCO/'
dataset_type = 'CocoDataset'
default_hooks = dict(
    checkpoint=dict(interval=1, type='CheckpointHook'),
    logger=dict(interval=50, type='LoggerHook'),
    param_scheduler=dict(type='ParamSchedulerHook'),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(type='DetVisualizationHook'))
default_scope = 'mmdet'
env_cfg = dict(
    cudnn_benchmark=False,
    dist_cfg=dict(backend='nccl'),
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
input_size = 300
launcher = 'none'
load_from = None
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 200
model = dict(
    backbone=dict(
        ceil_mode=True,
        depth=16,
        init_cfg=dict(
            checkpoint='open-mmlab://vgg16_caffe', type='Pretrained'),
        out_feature_indices=(
            22,
            34,
        ),
        out_indices=(
            3,
            4,
        ),
        type='SSDVGG',
        with_last_pool=False),
    bbox_head=dict(
        anchor_generator=dict(
            basesize_ratio_range=(
                0.15,
                0.9,
            ),
            input_size=300,
            ratios=[
                [
                    2,
                ],
                [
                    2,
                    3,
                ],
                [
                    2,
                    3,
                ],
                [
                    2,
                    3,
                ],
                [
                    2,
                ],
                [
                    2,
                ],
            ],
            scale_major=False,
            strides=[
                8,
                16,
                32,
                64,
                100,
                300,
            ],
            type='SSDAnchorGenerator'),
        bbox_coder=dict(
            target_means=[
                0.0,
                0.0,
                0.0,
                0.0,
            ],
            target_stds=[
                0.1,
                0.1,
                0.2,
                0.2,
            ],
            type='DeltaXYWHBBoxCoder'),
        in_channels=(
            512,
            1024,
            512,
            256,
            256,
            256,
        ),
        num_classes=6,
        type='SSDHead'),
    data_preprocessor=dict(
        bgr_to_rgb=True,
        mean=[
            123.675,
            116.28,
            103.53,
        ],
        pad_size_divisor=1,
        std=[
            1,
            1,
            1,
        ],
        type='DetDataPreprocessor'),
    neck=dict(
        in_channels=(
            512,
            1024,
        ),
        l2_norm_scale=20,
        level_paddings=(
            1,
            1,
            0,
            0,
        ),
        level_strides=(
            2,
            2,
            1,
            1,
        ),
        out_channels=(
            512,
            1024,
            512,
            256,
            256,
            256,
        ),
        type='SSDNeck'),
    test_cfg=dict(
        max_per_img=200,
        min_bbox_size=0,
        nms=dict(iou_threshold=0.45, type='nms'),
        nms_pre=1000,
        score_thr=0.02),
    train_cfg=dict(
        allowed_border=-1,
        assigner=dict(
            gt_max_assign_all=False,
            ignore_iof_thr=-1,
            min_pos_iou=0.0,
            neg_iou_thr=0.5,
            pos_iou_thr=0.5,
            type='MaxIoUAssigner'),
        debug=False,
        neg_pos_ratio=3,
        pos_weight=-1,
        sampler=dict(type='PseudoSampler'),
        smoothl1_beta=1.0),
    type='SingleStageDetector')
optim_wrapper = dict(
    optimizer=dict(lr=0.002, momentum=0.9, type='SGD', weight_decay=0.0005),
    type='OptimWrapper')
param_scheduler = [
    dict(
        begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'),
    dict(
        begin=0,
        by_epoch=True,
        end=24,
        gamma=0.1,
        milestones=[
            16,
            22,
        ],
        type='MultiStepLR'),
]
resume = False
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
    batch_size=1,
    dataset=dict(
        ann_file='/srv/samba/dingwenchao/SeaShips/COCO/annotations/test.json',
        backend_args=None,
        data_prefix=dict(img='test/'),
        data_root='/srv/samba/dingwenchao/SeaShips/COCO/',
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=False, scale=(
                300,
                300,
            ), type='Resize'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=2,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
    ann_file='/srv/samba/dingwenchao/SeaShips/COCO/annotations/val.json',
    backend_args=None,
    format_only=False,
    metric='bbox',
    type='CocoMetric')
test_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(keep_ratio=False, scale=(
        300,
        300,
    ), type='Resize'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'scale_factor',
        ),
        type='PackDetInputs'),
]
train_cfg = dict(max_epochs=200, type='EpochBasedTrainLoop', val_interval=1)
train_dataloader = dict(
    batch_sampler=None,
    batch_size=16,
    dataset=dict(
        dataset=dict(
            ann_file=
            '/srv/samba/dingwenchao/SeaShips/COCO/annotations/train.json',
            backend_args=None,
            data_prefix=dict(img='train/'),
            data_root='/srv/samba/dingwenchao/SeaShips/COCO/',
            filter_cfg=dict(filter_empty_gt=True, min_size=32),
            pipeline=[
                dict(backend_args=None, type='LoadImageFromFile'),
                dict(type='LoadAnnotations', with_bbox=True),
                dict(
                    mean=[
                        123.675,
                        116.28,
                        103.53,
                    ],
                    ratio_range=(
                        1,
                        4,
                    ),
                    to_rgb=True,
                    type='Expand'),
                dict(
                    min_crop_size=0.3,
                    min_ious=(
                        0.1,
                        0.3,
                        0.5,
                        0.7,
                        0.9,
                    ),
                    type='MinIoURandomCrop'),
                dict(keep_ratio=False, scale=(
                    300,
                    300,
                ), type='Resize'),
                dict(prob=0.5, type='RandomFlip'),
                dict(
                    brightness_delta=32,
                    contrast_range=(
                        0.5,
                        1.5,
                    ),
                    hue_delta=18,
                    saturation_range=(
                        0.5,
                        1.5,
                    ),
                    type='PhotoMetricDistortion'),
                dict(type='PackDetInputs'),
            ],
            type='CocoDataset'),
        times=5,
        type='RepeatDataset'),
    num_workers=8,
    persistent_workers=True,
    sampler=dict(shuffle=True, type='DefaultSampler'))
train_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        mean=[
            123.675,
            116.28,
            103.53,
        ],
        ratio_range=(
            1,
            4,
        ),
        to_rgb=True,
        type='Expand'),
    dict(
        min_crop_size=0.3,
        min_ious=(
            0.1,
            0.3,
            0.5,
            0.7,
            0.9,
        ),
        type='MinIoURandomCrop'),
    dict(keep_ratio=False, scale=(
        300,
        300,
    ), type='Resize'),
    dict(prob=0.5, type='RandomFlip'),
    dict(
        brightness_delta=32,
        contrast_range=(
            0.5,
            1.5,
        ),
        hue_delta=18,
        saturation_range=(
            0.5,
            1.5,
        ),
        type='PhotoMetricDistortion'),
    dict(type='PackDetInputs'),
]
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
    batch_size=16,
    dataset=dict(
        ann_file='/srv/samba/dingwenchao/SeaShips/COCO/annotations/val.json',
        backend_args=None,
        data_prefix=dict(img='val/'),
        data_root='/srv/samba/dingwenchao/SeaShips/COCO/',
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=False, scale=(
                300,
                300,
            ), type='Resize'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
    ann_file='/srv/samba/dingwenchao/SeaShips/COCO/annotations/val.json',
    backend_args=None,
    format_only=False,
    metric='bbox',
    type='CocoMetric')
vis_backends = [
    dict(type='LocalVisBackend'),
]
visualizer = dict(
    name='visualizer',
    type='DetLocalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
    ])
work_dir = './work_dirs/ssd300_coco'

09/12 15:44:10 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
09/12 15:44:10 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) RuntimeInfoHook                    
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
before_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DistSamplerSeedHook                
(NORMAL      ) NumClassCheckHook                  
 -------------------- 
before_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
(VERY_LOW    ) CheckInvalidLossHook               
 -------------------- 
after_train_epoch:
(NORMAL      ) IterTimerHook                      
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_val_epoch:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) NumClassCheckHook                  
 -------------------- 
before_val_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_val_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_val_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
after_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_test_epoch:
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_test_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_test_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
after_run:
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
Traceback (most recent call last):
  File "/srv/samba/dingwenchao/mmdetection/tools/train.py", line 106, in <module>
    main()
  File "/srv/samba/dingwenchao/mmdetection/tools/train.py", line 102, in main
    runner.train()
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/runner/runner.py", line 1703, in train
    self._train_loop = self.build_train_loop(
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/runner/runner.py", line 1495, in build_train_loop
    loop = LOOPS.build(
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/runner/loops.py", line 44, in __init__
    super().__init__(runner, dataloader)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/runner/base_loop.py", line 26, in __init__
    self.dataloader = runner.build_dataloader(
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/runner/runner.py", line 1353, in build_dataloader
    dataset = DATASETS.build(dataset_cfg)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/dataset/dataset_wrapper.py", line 211, in __init__
    self.dataset = DATASETS.build(dataset)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmdet/datasets/base_det_dataset.py", line 44, in __init__
    super().__init__(*args, **kwargs)
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/dataset/base_dataset.py", line 245, in __init__
    self.full_init()
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmdet/datasets/base_det_dataset.py", line 82, in full_init
    self.data_bytes, self.data_address = self._serialize_data()
  File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/dataset/base_dataset.py", line 765, in _serialize_data
    data_bytes = np.concatenate(data_list)
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate

[ZhouChen-start]

我出现这个问题是因为修改了数据集类别，使用python setup.py install重新编译一下

[ZhangYouyi]

File "/home/ding/.conda/envs/pt/lib/python3.9/site-packages/mmengine/dataset/base_dataset.py", line 765, in _serialize_data data_bytes = np.concatenate(data_list) 很可能是上面的中data_list为空所有致，data_list不能为空。可以通过调试一下data_list相关的初始化流程来进一步定位问题。

>>> import numpy as np
>>> np.concatenate([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: need at least one array to concatenate

[Doby6]

我出现这个问题是因为修改了数据集类别，使用python setup.py install重新编译一下

您好，请问是要在mmdetection文件夹第一个目录里进行重新setup.py的编译吗？

[ZhouChen-start]

我出现这个问题是因为修改了数据集类别，使用python setup.py install重新编译一下

您好，请问是要在mmdetection文件夹第一个目录里进行重新setup.py的编译吗？

就是在mmdetection这个文件夹下，你也可以搜一下mmdetection更改类别教程，里面会提到

[502dxceit]

I faced the same problem. yeah, I fixed the metainfo.classes in config file, which I take a spelling error. this classes name must be match to datasets' categories name, then it can be trained

[Doby6]

我出现这个问题是因为修改了数据集类别，使用python setup.py install重新编译一下

您好，请问是要在mmdetection文件夹第一个目录里进行重新setup.py的编译吗？

就是在mmdetection这个文件夹下，你也可以搜一下mmdetection更改类别教程，里面会提到

噢噢，好的，十分感谢。我还想问下我的是coco数据集也一样需要重新编译吗？因为我看了一些资料都是关于voc数据集的

[ZhouChen-start]

也是要的

---Original--- From: @.> Date: Tue, Oct 10, 2023 18:02 PM To: @.>; Cc: @.**@.>; Subject: Re: [open-mmlab/mmdetection] ValueError: need at least one array toconcatenate (Issue #10916)

我出现这个问题是因为修改了数据集类别，使用python setup.py install重新编译一下

您好，请问是要在mmdetection文件夹第一个目录里进行重新setup.py的编译吗？

就是在mmdetection这个文件夹下，你也可以搜一下mmdetection更改类别教程，里面会提到

噢噢，好的，十分感谢。我还想问下我的是coco数据集也一样需要重新编译吗？因为我看了一些资料都是关于voc数据集的

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[kpyon-w]

I have the same problem as you, the server failed to run the error array can not concat. I compared the local environment with the server environment, and found that the version of mmcv should be >=2.0.0 <2.1.0, and mmdet was called when the error was reported, but I did not have mmdet locally, so I uninstall mmdet and returned the mmcv version to 2.0.1, the problem was solved. Hope this will help you