Closed muzi2045 closed 4 years ago
check your data. Dimension out of range (expected to be in range of [-2, 1], but got 2)
it locate at backones(SpMiddleResNetModule which contain BN layer) occur this error, it mean I should check the forward pipeline in this part?
# input: # [41, 1600, 1408]
sparse_shape = np.array(input_shape[::-1]) + [1, 0, 0]
print("sparse shape:", sparse_shape)
coors = coors.int()
ret = spconv.SparseConvTensor(voxel_features, coors, sparse_shape, batch_size)
ret = self.middle_conv(ret)
ret = ret.dense()
N, C, D, H, W = ret.shape
ret = ret.view(N, C * D, H, W)
return ret
yes, 1d conv should be used here (sync bn layer)
the syncbn input tensor shape:
syncbn input shape: torch.Size([75100, 16])
syncbn input shape: torch.Size([76985, 16])
syncbn input shape: torch.Size([79617, 16])
syncbn input shape: torch.Size([81551, 16])
syncbn input shape: torch.Size([79113, 16])
syncbn input shape: torch.Size([68469, 16])
syncbn input shape: torch.Size([61096, 16])
syncbn input shape: torch.Size([77184, 16])
try to print spconv.SparseConvTensor shape info, but it can't, just use this:
print(" ret shape:", ret.dense().shape)
here is the output:
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
ret shape: torch.Size([4, 5, 41, 1008, 1008])
is there any way to debug the shape info in spconv.SparseTensor? @poodarchu
I guess you could run it using single gpu; so the problem is relative to BatchNorm Layer; you should use batchnrom 1d .
it's strange, in the code, the BN layer are set to batchnorm1d
## norm_cfg == None
if norm_cfg is None:
norm_cfg = dict(type="BN1d", eps=1e-3, momentum=0.01)
but the whole network info output is:
(backbone): SpMiddleResNetFHD(
(middle_conv): SparseSequential(
(0): SubMConv3d()
(1): SyncBatchNorm(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
(3): SparseBasicBlock(
(conv1): SubMConv3d()
(bn1): SyncBatchNorm(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(relu): ReLU()
(conv2): SubMConv3d()
(bn2): SyncBatchNorm(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
refer this issue in apex https://github.com/NVIDIA/apex/issues/194 @poodarchu
reinstall apex fix it
trying train CBGS in 8 GPU(2080ti), using the newest repo code, use follow code to start.
the error looks like happend in syncBN:
by the way, server environment are using pytorch 1.3.1 + CUDA 10.1 + python3.6 @poodarchu