search

microsoft / MeshTransformer

Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"
https://arxiv.org/abs/2012.09760
MIT License
609 stars 94 forks source link

v_shaped = torch.matmul(shapedirs, beta).view(-1, 6890, 3) + v_template RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` #56

Open Ngheissari opened 2 years ago

Ngheissari commented 2 years ago

While running

python -m torch.distributed.launch --nproc_per_node=8 metro/tools/run_metro_bodymesh.py for training:

Traceback (most recent call last): File "metro/tools/run_metro_bodymesh.py", line 717, in main(args) File "metro/tools/run_metro_bodymesh.py", line 711, in main run(args, train_dataloader, val_dataloader, _metro_network, smpl, mesh_sampler, renderer) File "metro/tools/run_metro_bodymesh.py", line 235, in run pred_camera, pred_3d_joints, pred_vertices_sub2, pred_vertices_sub, pred_vertices = METRO_model(images, smpl, mesh_sampler, meta_masks=meta_masks, is_train=True) File "....../miniconda3/envs/metro/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, *kwargs) File "...../MeshTransformer/metro/modeling/bert/modeling_metro.py", line 280, in forward features = featuresmeta_masks + constant_tensor(1-meta_masks)
File "....../miniconda3/envs/metro/lib/python3.7/site-packages/torch/tensor.py", line 394, in rsub return _C._VariableFunctions.rsub(self, other) RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 1.52 GiB already allocated; 9.00 MiB free; 1.55 GiB reserved in total by PyTorch) Traceback (most recent call last): File "metro/tools/run_metro_bodymesh.py", line 717, in main(args) File "metro/tools/run_metro_bodymesh.py", line 711, in main run(args, train_dataloader, val_dataloader, _metro_network, smpl, mesh_sampler, renderer) File "metro/tools/run_metro_bodymesh.py", line 221, in run gt_vertices_sub2 = mesh_sampler.downsample(gt_vertices, n1=0, n2=2) File .....MeshTransformer/metro/modeling/_smpl.py", line 272, in downsample y = spmm(self._D[j], y) File ".....MeshTransformer/metro/modeling/_smpl.py", line 172, in spmm return SparseMM.apply(sparse, dense) File "......MeshTransformer/metro/modeling/_smpl.py", line 161, in forward return torch.matmul(sparse, dense) RuntimeError: CUDA error: initialization error when calling cusparseCreate(handle) Traceback (most recent call last): File "metro/tools/run_metro_bodymesh.py", line 717, in main(args) File "metro/tools/run_metro_bodymesh.py", line 711, in main run(args, train_dataloader, val_dataloader, _metro_network, smpl, mesh_sampler, renderer) File "metro/tools/run_metro_bodymesh.py", line 220, in run gt_vertices = smpl(gt_pose, gt_betas) File "....../miniconda3/envs/metro/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, **kwargs) File ".....MeshTransformer/metro/modeling/_smpl.py", line 89, in forward v_shaped = torch.matmul(shapedirs, beta).view(-1, 6890, 3) + v_template RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

python=3.7 pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1

Also despite having 64 GB memory and reducing batch-size to 2 , still I get :

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 1.52 GiB already allocated; 9.00 MiB free; 1.55 GiB reserved in total by PyTorch)

torch.cuda.empty_cache() did not help.