RuntimeError: given chunk sizes don't sum up to the tensor's size (sum(chunk_sizes) == 48, but expected 1)

[hsj928]

[xjiao004]

The length of chunk size is the number of GPUs. And the batch size is the sum of chunks size.

[hsj928]

There is an new issue.

[Duankaiwen]

@hsj928 Add these codes above the dist = tag_mean.unsqueeze(1) - tag_mean.unsqueeze(2) in models/py_utils/kp_utils.py:

if len(tag_mean.size()) < 2:

      tag_mean = tag_mean.unsqueeze(0)`  

[Ostyk]

The length of chunk size is the number of GPUs. And the batch size is the sum of chunks size.

so if we use one GPU, we can only use chunk size of 1? I'm struggling to use more of the memory of my GPU in this way, even when I set the batch size really high

[Duankaiwen]

@Ostyk If you only have one GPU, modify the these codes in config/xxx.json: line4: "batch_size": xx, (where xx denotes the batch size in a GPU， it can be more than 1) line22: "chunk_sizes": [xx], (the 'chunk_sizes' is equal to the 'batch_size') I also recommend that you try the CPNDet (https://github.com/Duankaiwen/CPNDet). CPNDet is a version 2.0 of the CenterNet

[Ostyk]

Thanks for the quick answer. I'll check out the new network, and since it's also anchor free I can use it as the backbone just like CenterNet for FAIRMOT (reidentificiation).

[Ostyk]

still getting the error when I put batch_size == chunksizes "batch_size": 20, "chunk_sizes": [20], RuntimeError: given chunk sizes don't sum up to the tensor's size (sum(chunk_sizes) == 20, but expected 1) (scatter at ..\torch\csrc\cuda\comm.cpp:176)

[Duankaiwen]

@Ostyk Can i see your full log?

[Ostyk]

`

Traceback (most recent call last): File "train.py", line 210, in train(training_dbs, validation_db, args.start_iter) File "train.py", line 143, in train out_train = nnet.train(*training) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\nnet\py_factory.py", line 82, in train loss_kp = self.network(xs, ys) File "C:\Users\ai.admin\Anaconda3\envs\ObjectTracking\lib\site-packages\torch\nn\modules\module.py", line 547, in call result = self.forward(input, *kwargs) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\data_parallel.py", line 66, in forward inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids, self.chunk_sizes) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\data_parallel.py", line 77, in scatter return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim, chunk_sizes=self.chunk_sizes) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\scatter_gather.py", line 30, in scatter_kwargs inputs = scatter(inputs, target_gpus, dim, chunk_sizes) if inputs else [] File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\scatter_gather.py", line 25, in scatter return scatter_map(inputs) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\scatter_gather.py", line 18, in scatter_map return list(zip(map(scatter_map, obj))) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\scatter_gather.py", line 20, in scatter_map return list(map(list, zip(*map(scatter_map, obj)))) File "C:\Users\ai.admin\Desktop\ObjectTracking\CenterNet\models\py_utils\scatter_gather.py", line 15, in scatter_map return Scatter.apply(target_gpus, chunk_sizes, dim, obj) File "C:\Users\ai.admin\Anaconda3\envs\ObjectTracking\lib\site-packages\torch\nn\parallel_functions.py", line 89, in forward outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams) File "C:\Users\ai.admin\Anaconda3\envs\ObjectTracking\lib\site-packages\torch\cuda\comm.py", line 147, in scatter return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams)) RuntimeError: given chunk sizes don't sum up to the tensor's size (sum(chunk_sizes) == 10, but expected 1) (scatter at ..\torch\csrc\cuda\comm.cpp:176) (no backtrace available) `

I read that it might have something to do with nn.DataParallel but couldn't figure it out. I'd be grateful for any tips :)