search

open-mmlab / mmskeleton

A OpenMMLAB toolbox for human pose estimation, skeleton-based action recognition, and action synthesis.
Apache License 2.0
2.92k stars 1.03k forks source link

RuntimeError: CUDA error: invalid argument #201

Open ITdoubleseven opened 5 years ago

ITdoubleseven commented 5 years ago

hello,when I run "python main.py recognition -c config/st_gcn/kinetics-skeleton/test.yaml",the following error occurred: [06.20.19|20:28:58] Load weights from ./models/st_gcn.kinetics.pt. [06.20.19|20:28:58] Load weights [A]. [06.20.19|20:28:58] Load weights [data_bn.weight]. [06.20.19|20:28:58] Load weights [data_bn.bias]. [06.20.19|20:28:58] Load weights [data_bn.running_mean]. [06.20.19|20:28:58] Load weights [data_bn.running_var]. [06.20.19|20:28:58] Load weights [data_bn.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.0.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.1.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.2.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.3.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.1.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.1.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.1.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.1.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.4.residual.1.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.5.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.6.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.1.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.1.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.1.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.1.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.7.residual.1.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.8.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.gcn.conv.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.gcn.conv.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.0.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.0.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.0.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.0.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.0.num_batches_tracked]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.2.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.2.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.3.weight]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.3.bias]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.3.running_mean]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.3.running_var]. [06.20.19|20:28:58] Load weights [st_gcn_networks.9.tcn.3.num_batches_tracked]. [06.20.19|20:28:58] Load weights [edge_importance.0]. [06.20.19|20:28:58] Load weights [edge_importance.1]. [06.20.19|20:28:58] Load weights [edge_importance.2]. [06.20.19|20:28:58] Load weights [edge_importance.3]. [06.20.19|20:28:58] Load weights [edge_importance.4]. [06.20.19|20:28:58] Load weights [edge_importance.5]. [06.20.19|20:28:58] Load weights [edge_importance.6]. [06.20.19|20:28:58] Load weights [edge_importance.7]. [06.20.19|20:28:58] Load weights [edge_importance.8]. [06.20.19|20:28:58] Load weights [edge_importance.9]. [06.20.19|20:28:58] Load weights [fcn.weight]. [06.20.19|20:28:58] Load weights [fcn.bias]. [06.20.19|20:28:58] Parameters: {'work_dir': './work_dir/tmp', 'config': 'config/st_gcn/kinetics-skeleton/test.yaml', 'phase': 'test', 'save_result': False, 'start_epoch': 0, 'num_epoch': 80, 'use_gpu': True, 'device': 0, 'log_interval': 100, 'save_interval': 10, 'eval_interval': 5, 'save_log': True, 'print_log': True, 'pavi_log': False, 'feeder': 'feeder.feeder.Feeder', 'num_worker': 4, 'train_feeder_args': {'debug': False}, 'test_feeder_args': {'data_path': './data/Kinetics/kinetics-skeleton/val_data.npy', 'label_path': './data/Kinetics/kinetics-skeleton/val_label.pkl'}, 'batch_size': 256, 'test_batch_size': 64, 'debug': False, 'model': 'net.st_gcn.Model', 'model_args': {'in_channels': 3, 'num_class': 400, 'edge_importance_weighting': True, 'graph_args': {'layout': 'openpose', 'strategy': 'spatial'}}, 'weights': './models/st_gcn.kinetics.pt', 'ignore_weights': [], 'show_topk': [1, 5], 'base_lr': 0.01, 'step': [], 'optimizer': 'SGD', 'nesterov': True, 'weight_decay': 0.0001}

[06.20.19|20:28:58] Model: net.st_gcn.Model. [06.20.19|20:28:58] Weights: ./models/st_gcn.kinetics.pt. [06.20.19|20:28:58] Evaluation Start: Traceback (most recent call last): File "main.py", line 31, in p.start() File "/home/njuciairs/HAQ/st-gcn/processor/processor.py", line 139, in start self.test() File "/home/njuciairs/HAQ/st-gcn/processor/recognition.py", line 126, in test output = self.model(data) File "/home/njuciairs/InstallTools/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/njuciairs/HAQ/st-gcn/net/st_gcn.py", line 77, in forward x = x.permute(0, 1, 3, 4, 2).contiguous() RuntimeError: CUDA error: invalid argument why this occurred and how to solve this problem? looking forward to your reply--

ITdoubleseven commented 5 years ago

whether add "--device 0" or not is the same.

wardpeng commented 5 years ago

Hi, @ITdoubleseven I have same error on my sider.

Do you solve it now? Thanks a lot.

When I run cmd 'python main.py demo --openpose /home/ward/Downloads/openpose/build --video /home/ward/Downloads/st-gcn/resource/media/ta_chi.mp4 --device 0'

Logs:

Network forwad... Traceback (most recent call last): File "main.py", line 31, in p.start() File "/home/ward/Downloads/st-gcn/processor/demo.py", line 70, in start output, feature = self.model.extract_feature(data) File "/home/ward/Downloads/st-gcn/net/st_gcn.py", line 102, in extract_feature x = x.permute(0, 1, 3, 4, 2).contiguous() RuntimeError: CUDA error: invalid argument

And my environments: Ubuntu 18.04 CUDA 9.2 (try to switch to 10.2, but same issue ) OpenPose Laterest version to July 2.(Already run openpose test successfully locally)

chaurasiat commented 5 years ago

Yeah I am also getting the same error when i try to run demo Network forwad... Traceback (most recent call last): File "main.py", line 31, in p.start() File "/home/ubuntu/trishla/st-gcn/processor/demo.py", line 70, in start output, feature = self.model.extract_feature(data) File "/home/ubuntu/trishla/st-gcn/net/st_gcn.py", line 102, in extract_feature x = x.permute(0, 1, 3, 4, 2).contiguous() RuntimeError: CUDA error: invalid argument Environment: ubuntu:16.04 CUDA 10.0(same error with cuda 10 and cuda 10.1)

Hunkzer commented 5 years ago

You need to install PyTorch 1.0 instead of 1.1 or 1.2.

223