dataloader does not work correctly

[SkyStephenTao]

when I run

data_loader = DataLoader(
            dataset=dataset,
            sampler=sampler if batch_sampler is None else None,
            batch_sampler=batch_sampler,
            collate_fn=collate_fn,
            worker_init_fn=init_fn,
            **dataloader_cfg)

I try to get batch data from data_loader. My original data size is all (256,256), and i set batch size=6, but batch i get from data_loader does not have the correct size(256,256). I check source to find the problem. And show the corresponding code as follows:

    def fetch(self, possibly_batched_index):
        if self.auto_collation:
            data = [self.dataset[idx] for idx in possibly_batched_index]
        else:
            data = self.dataset[possibly_batched_index]
        return self.collate_fn(data)

the data[1].size=（3,170,170），while the other data elements' size = (3,256,256)

[xiexinch]

Hi @SkyStephenTao, Could you show me where the source code you post is?

[SkyStephenTao]

Thank you for your replay. The problem had been resolved. The problem is caused by my data preprocess. I execute resize and crop preprocessing.

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2023年05月17日 11:21 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [open-mmlab/mmsegmentation] dataloader does not work correctly (Issue #3014) |

Hi @SkyStephenTao, Could you show me where the source code you post is?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[haiw2sm]

I have the same the very similar problem ....... How to solved it? Why i cannot solve it yet.

Environment

python==3.10.3
pytorch==1.13.1
cuda==11.7
mmcv-full==1.7.2
mmsegmentation==0.30.0

Problem Code Entry

    data_loader = dataloader(
        dataset,
        batch_size=batch_size,
        sampler=sampler,
        num_workers=num_workers,
        collate_fn=partial(collate, samples_per_gpu=samples_per_gpu),
        pin_memory=pin_memory,
        shuffle=shuffle,
        worker_init_fn=init_fn,
        drop_last=drop_last,
        **kwargs)

Dataset Module In mmseg Registried

Include among these, dataset is in the train mode, the mmseg dataset module registried is

    train=dict(
        type='RepeatDataset',
        times=500,
        dataset=dict(
            type="ACDCDataset",
            data_root=cityscapes_root,
            img_dir='leftImg8bit/train',
            ann_dir='gtFine/train',
            pipeline=train_pipeline
        )),

Error Log

Traceback (most recent call last):
  File "C:\Users\spame\.pyenv\pyenv-win\versions\3.10.3\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\spame\.pyenv\pyenv-win\versions\3.10.3\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\pro\os\cv\CTTAandVPTTA\CttaRegFormer2MaskFormer\tools\train.py", line 168, in <module>
    main()
  File "D:\pro\os\cv\CTTAandVPTTA\CttaRegFormer2MaskFormer\tools\train.py", line 157, in main
    train_segmentor(
  File "D:\pro\os\cv\CTTAandVPTTA\CttaRegFormer2MaskFormer\mmseg\apis\train.py", line 115, in train_segmentor
    runner.run(data_loaders, cfg.workflow)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\mmcv\runner\iter_based_runner.py", line 144, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\mmcv\runner\iter_based_runner.py", line 61, in train        
    data_batch = next(data_loader)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\mmcv\runner\iter_based_runner.py", line 34, in __next__     
    data = next(self.iter_loader)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\dataloader.py", line 628, in __next__      
    data = self._next_data()
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data   
    return self._process_data(data)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
    data.reraise()
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\_utils.py", line 543, in reraise
    raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\_utils\worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 61, in fetch        
    return self.collate_fn(data)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\mmcv\parallel\collate.py", line 84, in collate
    return default_collate(batch)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\_utils\collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "D:\pro\os\cv\CTTAandVPTTA\formervenv\lib\site-packages\torch\utils\data\_utils\collate.py", line 151, in collate   
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'mmseg.datasets.dataset_wrappers.RepeatDataset'>

Iteration make the batch object elements are the dataset module registered dont match the collate.default_collate.

Thrown Exception Code Description, Location

And the collate function in the mmcv.parallel.collate.py (Location) only detect the following dataset type Sequence, DataContainer, Mapping

Replenishment

RepeatDataset and ACDCDataset(Customizable) has the__getitem__ magic function