hello @dutran , i finetuned the R2plus1D network pretrained by Kinetics and this error occured. could you help me solve this please, thank you.
INFO:model_builder:creating r2plus1d, depth=18...
INFO:video_model:Number of middle filters: 144
INFO:video_model:Number of middle filters: 144
INFO:video_model:Number of middle filters: 144
INFO:video_model:Number of middle filters: 144
INFO:video_model:Number of middle filters: 230
INFO:video_model:Number of middle filters: 288
INFO:video_model:Number of middle filters: 288
INFO:video_model:Number of middle filters: 288
INFO:video_model:Number of middle filters: 460
INFO:video_model:Number of middle filters: 576
INFO:video_model:Number of middle filters: 576
INFO:video_model:Number of middle filters: 576
INFO:video_model:Number of middle filters: 921
INFO:video_model:Number of middle filters: 1152
INFO:video_model:Number of middle filters: 1152
INFO:video_model:Number of middle filters: 1152
INFO:data_parallel_model:Parameter update function not defined --> only forward
INFO:model_loader:copying conv1_middle_w to gpu_0/conv1_middle_w
INFO:model_loader:copying conv1_middle_spatbn_relu_s to gpu_0/conv1_middle_spatbn_relu_s
INFO:model_loader:copying conv1_middle_spatbn_relu_b to gpu_0/conv1_middle_spatbn_relu_b
INFO:model_loader:copying conv1_w to gpu_0/conv1_w
INFO:model_loader:copying conv1_spatbn_relu_s to gpu_0/conv1_spatbn_relu_s
INFO:model_loader:copying conv1_spatbn_relu_b to gpu_0/conv1_spatbn_relu_b
INFO:model_loader:copying comp_0_conv_1_middle_w to gpu_0/comp_0_conv_1_middle_w
INFO:model_loader:copying comp_0_spatbn_1_middle_s to gpu_0/comp_0_spatbn_1_middle_s
INFO:model_loader:copying comp_0_spatbn_1_middle_b to gpu_0/comp_0_spatbn_1_middle_b
INFO:model_loader:copying comp_0_conv_1_w to gpu_0/comp_0_conv_1_w
INFO:model_loader:copying comp_0_spatbn_1_s to gpu_0/comp_0_spatbn_1_s
INFO:model_loader:copying comp_0_spatbn_1_b to gpu_0/comp_0_spatbn_1_b
INFO:model_loader:copying comp_0_conv_2_middle_w to gpu_0/comp_0_conv_2_middle_w
INFO:model_loader:copying comp_0_spatbn_2_middle_s to gpu_0/comp_0_spatbn_2_middle_s
INFO:model_loader:copying comp_0_spatbn_2_middle_b to gpu_0/comp_0_spatbn_2_middle_b
INFO:model_loader:copying comp_0_conv_2_w to gpu_0/comp_0_conv_2_w
INFO:model_loader:copying comp_0_spatbn_2_s to gpu_0/comp_0_spatbn_2_s
INFO:model_loader:copying comp_0_spatbn_2_b to gpu_0/comp_0_spatbn_2_b
INFO:model_loader:copying comp_1_conv_1_middle_w to gpu_0/comp_1_conv_1_middle_w
INFO:model_loader:copying comp_1_spatbn_1_middle_s to gpu_0/comp_1_spatbn_1_middle_s
INFO:model_loader:copying comp_1_spatbn_1_middle_b to gpu_0/comp_1_spatbn_1_middle_b
INFO:model_loader:copying comp_1_conv_1_w to gpu_0/comp_1_conv_1_w
INFO:model_loader:copying comp_1_spatbn_1_s to gpu_0/comp_1_spatbn_1_s
INFO:model_loader:copying comp_1_spatbn_1_b to gpu_0/comp_1_spatbn_1_b
INFO:model_loader:copying comp_1_conv_2_middle_w to gpu_0/comp_1_conv_2_middle_w
INFO:model_loader:copying comp_1_spatbn_2_middle_s to gpu_0/comp_1_spatbn_2_middle_s
INFO:model_loader:copying comp_1_spatbn_2_middle_b to gpu_0/comp_1_spatbn_2_middle_b
INFO:model_loader:copying comp_1_conv_2_w to gpu_0/comp_1_conv_2_w
INFO:model_loader:copying comp_1_spatbn_2_s to gpu_0/comp_1_spatbn_2_s
INFO:model_loader:copying comp_1_spatbn_2_b to gpu_0/comp_1_spatbn_2_b
INFO:model_loader:found comp_2_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_2_conv_1_w but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_s but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_b but blob shape do not match
INFO:model_loader:found comp_2_conv_2_middle_w but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_middle_s but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_middle_b but blob shape do not match
INFO:model_loader:found comp_2_conv_2_w but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_s but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_b but blob shape do not match
INFO:model_loader:shortcut_projection_2_w not found
INFO:model_loader:shortcut_projection_2_spatbn_s not found
INFO:model_loader:shortcut_projection_2_spatbn_b not found
INFO:model_loader:found comp_3_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_3_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_3_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_3_conv_1_w but blob shape do not match
INFO:model_loader:copying comp_3_spatbn_1_s to gpu_0/comp_3_spatbn_1_s
INFO:model_loader:copying comp_3_spatbn_1_b to gpu_0/comp_3_spatbn_1_b
INFO:model_loader:copying comp_3_conv_2_middle_w to gpu_0/comp_3_conv_2_middle_w
INFO:model_loader:copying comp_3_spatbn_2_middle_s to gpu_0/comp_3_spatbn_2_middle_s
INFO:model_loader:copying comp_3_spatbn_2_middle_b to gpu_0/comp_3_spatbn_2_middle_b
INFO:model_loader:copying comp_3_conv_2_w to gpu_0/comp_3_conv_2_w
INFO:model_loader:copying comp_3_spatbn_2_s to gpu_0/comp_3_spatbn_2_s
INFO:model_loader:copying comp_3_spatbn_2_b to gpu_0/comp_3_spatbn_2_b
INFO:model_loader:found comp_4_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_4_conv_1_w but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_s but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_b but blob shape do not match
INFO:model_loader:found comp_4_conv_2_middle_w but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_middle_s but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_middle_b but blob shape do not match
INFO:model_loader:found comp_4_conv_2_w but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_s but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_b but blob shape do not match
INFO:model_loader:shortcut_projection_4_w not found
INFO:model_loader:shortcut_projection_4_spatbn_s not found
INFO:model_loader:shortcut_projection_4_spatbn_b not found
INFO:model_loader:found comp_5_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_5_conv_1_w but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_s but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_b but blob shape do not match
INFO:model_loader:found comp_5_conv_2_middle_w but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_middle_s but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_middle_b but blob shape do not match
INFO:model_loader:found comp_5_conv_2_w but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_s but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_b but blob shape do not match
INFO:model_loader:found comp_6_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_6_conv_1_w but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_s but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_b but blob shape do not match
INFO:model_loader:found comp_6_conv_2_middle_w but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_middle_s but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_middle_b but blob shape do not match
INFO:model_loader:found comp_6_conv_2_w but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_s but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_b but blob shape do not match
INFO:model_loader:shortcut_projection_6_w not found
INFO:model_loader:shortcut_projection_6_spatbn_s not found
INFO:model_loader:shortcut_projection_6_spatbn_b not found
INFO:model_loader:found comp_7_conv_1_middle_w but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_middle_s but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_middle_b but blob shape do not match
INFO:model_loader:found comp_7_conv_1_w but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_s but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_b but blob shape do not match
INFO:model_loader:found comp_7_conv_2_middle_w but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_middle_s but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_middle_b but blob shape do not match
INFO:model_loader:found comp_7_conv_2_w but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_s but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_b but blob shape do not match
INFO:model_loader:last_out_L5_w not found
INFO:model_loader:last_out_L5_b not found
INFO:model_loader:copying conv1_middle_spatbn_relu_rm to gpu_0/conv1_middle_spatbn_relu_rm
INFO:model_loader:copying conv1_middle_spatbn_relu_riv to gpu_0/conv1_middle_spatbn_relu_riv
INFO:model_loader:copying conv1_spatbn_relu_rm to gpu_0/conv1_spatbn_relu_rm
INFO:model_loader:copying conv1_spatbn_relu_riv to gpu_0/conv1_spatbn_relu_riv
INFO:model_loader:copying comp_0_spatbn_1_middle_rm to gpu_0/comp_0_spatbn_1_middle_rm
INFO:model_loader:copying comp_0_spatbn_1_middle_riv to gpu_0/comp_0_spatbn_1_middle_riv
INFO:model_loader:copying comp_0_spatbn_1_rm to gpu_0/comp_0_spatbn_1_rm
INFO:model_loader:copying comp_0_spatbn_1_riv to gpu_0/comp_0_spatbn_1_riv
INFO:model_loader:copying comp_0_spatbn_2_middle_rm to gpu_0/comp_0_spatbn_2_middle_rm
INFO:model_loader:copying comp_0_spatbn_2_middle_riv to gpu_0/comp_0_spatbn_2_middle_riv
INFO:model_loader:copying comp_0_spatbn_2_rm to gpu_0/comp_0_spatbn_2_rm
INFO:model_loader:copying comp_0_spatbn_2_riv to gpu_0/comp_0_spatbn_2_riv
INFO:model_loader:copying comp_1_spatbn_1_middle_rm to gpu_0/comp_1_spatbn_1_middle_rm
INFO:model_loader:copying comp_1_spatbn_1_middle_riv to gpu_0/comp_1_spatbn_1_middle_riv
INFO:model_loader:copying comp_1_spatbn_1_rm to gpu_0/comp_1_spatbn_1_rm
INFO:model_loader:copying comp_1_spatbn_1_riv to gpu_0/comp_1_spatbn_1_riv
INFO:model_loader:copying comp_1_spatbn_2_middle_rm to gpu_0/comp_1_spatbn_2_middle_rm
INFO:model_loader:copying comp_1_spatbn_2_middle_riv to gpu_0/comp_1_spatbn_2_middle_riv
INFO:model_loader:copying comp_1_spatbn_2_rm to gpu_0/comp_1_spatbn_2_rm
INFO:model_loader:copying comp_1_spatbn_2_riv to gpu_0/comp_1_spatbn_2_riv
INFO:model_loader:found comp_2_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_rm but blob shape do not match
INFO:model_loader:found comp_2_spatbn_1_riv but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_middle_rm but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_middle_riv but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_rm but blob shape do not match
INFO:model_loader:found comp_2_spatbn_2_riv but blob shape do not match
INFO:model_loader:shortcut_projection_2_spatbn_rm not found
INFO:model_loader:shortcut_projection_2_spatbn_riv not found
INFO:model_loader:found comp_3_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_3_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:copying comp_3_spatbn_1_rm to gpu_0/comp_3_spatbn_1_rm
INFO:model_loader:copying comp_3_spatbn_1_riv to gpu_0/comp_3_spatbn_1_riv
INFO:model_loader:copying comp_3_spatbn_2_middle_rm to gpu_0/comp_3_spatbn_2_middle_rm
INFO:model_loader:copying comp_3_spatbn_2_middle_riv to gpu_0/comp_3_spatbn_2_middle_riv
INFO:model_loader:copying comp_3_spatbn_2_rm to gpu_0/comp_3_spatbn_2_rm
INFO:model_loader:copying comp_3_spatbn_2_riv to gpu_0/comp_3_spatbn_2_riv
INFO:model_loader:found comp_4_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_rm but blob shape do not match
INFO:model_loader:found comp_4_spatbn_1_riv but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_middle_rm but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_middle_riv but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_rm but blob shape do not match
INFO:model_loader:found comp_4_spatbn_2_riv but blob shape do not match
INFO:model_loader:shortcut_projection_4_spatbn_rm not found
INFO:model_loader:shortcut_projection_4_spatbn_riv not found
INFO:model_loader:found comp_5_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_rm but blob shape do not match
INFO:model_loader:found comp_5_spatbn_1_riv but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_middle_rm but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_middle_riv but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_rm but blob shape do not match
INFO:model_loader:found comp_5_spatbn_2_riv but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_rm but blob shape do not match
INFO:model_loader:found comp_6_spatbn_1_riv but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_middle_rm but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_middle_riv but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_rm but blob shape do not match
INFO:model_loader:found comp_6_spatbn_2_riv but blob shape do not match
INFO:model_loader:shortcut_projection_6_spatbn_rm not found
INFO:model_loader:shortcut_projection_6_spatbn_riv not found
INFO:model_loader:found comp_7_spatbn_1_middle_rm but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_middle_riv but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_rm but blob shape do not match
INFO:model_loader:found comp_7_spatbn_1_riv but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_middle_rm but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_middle_riv but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_rm but blob shape do not match
INFO:model_loader:found comp_7_spatbn_2_riv but blob shape do not match
INFO:data_parallel_model:Creating checkpoint synchronization net
INFO:data_parallel_model:Run checkpoint net
INFO:train_net:Starting epoch 0/8
[swscaler @ 0x7d1aa4175d20] Warning: data is not aligned! This can lead to a speed loss
[E net_async_base.cc:382] [enforce fail at conv_op_cudnn.cc:520] filter.dim() >= 3 && filter.dim() <= 5.
Error from operator:
input: "gpu_0/comp_1_sum_2" input: "gpu_0/comp_2_conv_1_middle_w" output: "gpu_0/comp_2_conv_1_middle" name: "" type: "Conv" arg { name: "kernels" ints: 1 ints: 3 ints: 3 } arg { name: "ws_nbytes_limit" i: 67108864 } arg { name: "exhaustive_search" i: 1 } arg { name: "strides" ints: 1 ints: 2 ints: 2 } arg { name: "pads" ints: 0 ints: 1 ints: 1 ints: 0 ints: 1 ints: 1 } arg { name: "order" s: "NCHW" } device_option { device_type: 1 device_id: 0 } engine: "CUDNN"frame #0: c10::ThrowEnforceNotMet(char const, int, char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void const) + 0x78 (0x7f1bb3a76178 in /usr/local/lib/libc10.so)
frame #1: + 0x3b11ba5 (0x7f1b780abba5 in /usr/local/lib/libtorch.so)
frame #2: + 0x3b00e08 (0x7f1b7809ae08 in /usr/local/lib/libtorch.so)
frame #3: + 0x3a7df9d (0x7f1b78017f9d in /usr/local/lib/libtorch.so)
frame #4: caffe2::AsyncNetBase::run(int, int) + 0x118 (0x7f1b768d2588 in /usr/local/lib/libtorch.so)
frame #5: + 0x233ee2a (0x7f1b768d8e2a in /usr/local/lib/libtorch.so)
frame #6: c10::ThreadPool::main_loop(unsigned long) + 0x2b3 (0x7f1bb3a704a3 in /usr/local/lib/libc10.so)
frame #7: + 0xbd66f (0x7f1b8c2bd66f in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x76db (0x7f1bb837b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f1bb86b488f in /lib/x86_64-linux-gnu/libc.so.6)
, op Conv
[E net_async_base.cc:134] Rethrowing exception from the run of 'r2plus1d_train'
WARNING:caffe2.python.workspace:Original python traceback for operator 34 in network r2plus1d_train in exception above (most recent call last):
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 501, in
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 496, in main
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 280, in Train
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/data_parallel_model.py", line 39, in Parallelize_GPU
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/data_parallel_model.py", line 237, in Parallelize
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 217, in create_model_ops
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/model_builder.py", line 128, in build_model
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/r3d_model.py", line 99, in create_model
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/r3d_model.py", line 192, in create_r3d
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/video_model.py", line 249, in add_simple_block
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/video_model.py", line 67, in add_conv
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/cnn.py", line 86, in ConvNd
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/brew.py", line 108, in scope_wrapper
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/helpers/conv.py", line 164, in conv_nd
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/helpers/conv.py", line 123, in _ConvBase
Traceback (most recent call last):
File "/usr/VMZ-master/tools/train_net.py", line 501, in
main()
File "/usr/VMZ-master/tools/train_net.py", line 496, in main
Train(args)
File "/usr/VMZ-master/tools/train_net.py", line 388, in Train
explog
File "/usr/VMZ-master/tools/train_net.py", line 123, in RunEpoch
workspace.RunNet(train_model.net.Proto().name)
File "/root/pytorch/build/caffe2/python/workspace.py", line 254, in RunNet
StringifyNetName(name), num_iter, allow_fail,
File "/root/pytorch/build/caffe2/python/workspace.py", line 215, in CallWithExceptionIntercept
return func(args, *kwargs)
RuntimeError: [enforce fail at conv_op_cudnn.cc:520] filter.dim() >= 3 && filter.dim() <= 5.
Error from operator:
input: "gpu_0/comp_1_sum_2" input: "gpu_0/comp_2_conv_1_middle_w" output: "gpu_0/comp_2_conv_1_middle" name: "" type: "Conv" arg { name: "kernels" ints: 1 ints: 3 ints: 3 } arg { name: "ws_nbytes_limit" i: 67108864 } arg { name: "exhaustive_search" i: 1 } arg { name: "strides" ints: 1 ints: 2 ints: 2 } arg { name: "pads" ints: 0 ints: 1 ints: 1 ints: 0 ints: 1 ints: 1 } arg { name: "order" s: "NCHW" } device_option { device_type: 1 device_id: 0 } engine: "CUDNN"frame #0: c10::ThrowEnforceNotMet(char const, int, char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void const) + 0x78 (0x7f1bb3a76178 in /usr/local/lib/libc10.so)
frame #1: + 0x3b11ba5 (0x7f1b780abba5 in /usr/local/lib/libtorch.so)
frame #2: + 0x3b00e08 (0x7f1b7809ae08 in /usr/local/lib/libtorch.so)
frame #3: + 0x3a7df9d (0x7f1b78017f9d in /usr/local/lib/libtorch.so)
frame #4: caffe2::AsyncNetBase::run(int, int) + 0x118 (0x7f1b768d2588 in /usr/local/lib/libtorch.so)
frame #5: + 0x233ee2a (0x7f1b768d8e2a in /usr/local/lib/libtorch.so)
frame #6: c10::ThreadPool::main_loop(unsigned long) + 0x2b3 (0x7f1bb3a704a3 in /usr/local/lib/libc10.so)
frame #7: + 0xbd66f (0x7f1b8c2bd66f in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x76db (0x7f1bb837b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f1bb86b488f in /lib/x86_64-linux-gnu/libc.so.6)
hello @dutran , i finetuned the R2plus1D network pretrained by Kinetics and this error occured. could you help me solve this please, thank you.
INFO:model_builder:creating r2plus1d, depth=18... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Parameter update function not defined --> only forward INFO:model_loader:copying conv1_middle_w to gpu_0/conv1_middle_w INFO:model_loader:copying conv1_middle_spatbn_relu_s to gpu_0/conv1_middle_spatbn_relu_s INFO:model_loader:copying conv1_middle_spatbn_relu_b to gpu_0/conv1_middle_spatbn_relu_b INFO:model_loader:copying conv1_w to gpu_0/conv1_w INFO:model_loader:copying conv1_spatbn_relu_s to gpu_0/conv1_spatbn_relu_s INFO:model_loader:copying conv1_spatbn_relu_b to gpu_0/conv1_spatbn_relu_b INFO:model_loader:copying comp_0_conv_1_middle_w to gpu_0/comp_0_conv_1_middle_w INFO:model_loader:copying comp_0_spatbn_1_middle_s to gpu_0/comp_0_spatbn_1_middle_s INFO:model_loader:copying comp_0_spatbn_1_middle_b to gpu_0/comp_0_spatbn_1_middle_b INFO:model_loader:copying comp_0_conv_1_w to gpu_0/comp_0_conv_1_w INFO:model_loader:copying comp_0_spatbn_1_s to gpu_0/comp_0_spatbn_1_s INFO:model_loader:copying comp_0_spatbn_1_b to gpu_0/comp_0_spatbn_1_b INFO:model_loader:copying comp_0_conv_2_middle_w to gpu_0/comp_0_conv_2_middle_w INFO:model_loader:copying comp_0_spatbn_2_middle_s to gpu_0/comp_0_spatbn_2_middle_s INFO:model_loader:copying comp_0_spatbn_2_middle_b to gpu_0/comp_0_spatbn_2_middle_b INFO:model_loader:copying comp_0_conv_2_w to gpu_0/comp_0_conv_2_w INFO:model_loader:copying comp_0_spatbn_2_s to gpu_0/comp_0_spatbn_2_s INFO:model_loader:copying comp_0_spatbn_2_b to gpu_0/comp_0_spatbn_2_b INFO:model_loader:copying comp_1_conv_1_middle_w to gpu_0/comp_1_conv_1_middle_w INFO:model_loader:copying comp_1_spatbn_1_middle_s to gpu_0/comp_1_spatbn_1_middle_s INFO:model_loader:copying comp_1_spatbn_1_middle_b to gpu_0/comp_1_spatbn_1_middle_b INFO:model_loader:copying comp_1_conv_1_w to gpu_0/comp_1_conv_1_w INFO:model_loader:copying comp_1_spatbn_1_s to gpu_0/comp_1_spatbn_1_s INFO:model_loader:copying comp_1_spatbn_1_b to gpu_0/comp_1_spatbn_1_b INFO:model_loader:copying comp_1_conv_2_middle_w to gpu_0/comp_1_conv_2_middle_w INFO:model_loader:copying comp_1_spatbn_2_middle_s to gpu_0/comp_1_spatbn_2_middle_s INFO:model_loader:copying comp_1_spatbn_2_middle_b to gpu_0/comp_1_spatbn_2_middle_b INFO:model_loader:copying comp_1_conv_2_w to gpu_0/comp_1_conv_2_w INFO:model_loader:copying comp_1_spatbn_2_s to gpu_0/comp_1_spatbn_2_s INFO:model_loader:copying comp_1_spatbn_2_b to gpu_0/comp_1_spatbn_2_b INFO:model_loader:found comp_2_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_2_conv_1_w but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_s but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_b but blob shape do not match INFO:model_loader:found comp_2_conv_2_middle_w but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_middle_s but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_middle_b but blob shape do not match INFO:model_loader:found comp_2_conv_2_w but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_s but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_b but blob shape do not match INFO:model_loader:shortcut_projection_2_w not found INFO:model_loader:shortcut_projection_2_spatbn_s not found INFO:model_loader:shortcut_projection_2_spatbn_b not found INFO:model_loader:found comp_3_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_3_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_3_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_3_conv_1_w but blob shape do not match INFO:model_loader:copying comp_3_spatbn_1_s to gpu_0/comp_3_spatbn_1_s INFO:model_loader:copying comp_3_spatbn_1_b to gpu_0/comp_3_spatbn_1_b INFO:model_loader:copying comp_3_conv_2_middle_w to gpu_0/comp_3_conv_2_middle_w INFO:model_loader:copying comp_3_spatbn_2_middle_s to gpu_0/comp_3_spatbn_2_middle_s INFO:model_loader:copying comp_3_spatbn_2_middle_b to gpu_0/comp_3_spatbn_2_middle_b INFO:model_loader:copying comp_3_conv_2_w to gpu_0/comp_3_conv_2_w INFO:model_loader:copying comp_3_spatbn_2_s to gpu_0/comp_3_spatbn_2_s INFO:model_loader:copying comp_3_spatbn_2_b to gpu_0/comp_3_spatbn_2_b INFO:model_loader:found comp_4_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_4_conv_1_w but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_s but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_b but blob shape do not match INFO:model_loader:found comp_4_conv_2_middle_w but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_middle_s but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_middle_b but blob shape do not match INFO:model_loader:found comp_4_conv_2_w but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_s but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_b but blob shape do not match INFO:model_loader:shortcut_projection_4_w not found INFO:model_loader:shortcut_projection_4_spatbn_s not found INFO:model_loader:shortcut_projection_4_spatbn_b not found INFO:model_loader:found comp_5_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_5_conv_1_w but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_s but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_b but blob shape do not match INFO:model_loader:found comp_5_conv_2_middle_w but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_middle_s but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_middle_b but blob shape do not match INFO:model_loader:found comp_5_conv_2_w but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_s but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_b but blob shape do not match INFO:model_loader:found comp_6_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_6_conv_1_w but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_s but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_b but blob shape do not match INFO:model_loader:found comp_6_conv_2_middle_w but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_middle_s but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_middle_b but blob shape do not match INFO:model_loader:found comp_6_conv_2_w but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_s but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_b but blob shape do not match INFO:model_loader:shortcut_projection_6_w not found INFO:model_loader:shortcut_projection_6_spatbn_s not found INFO:model_loader:shortcut_projection_6_spatbn_b not found INFO:model_loader:found comp_7_conv_1_middle_w but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_middle_s but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_middle_b but blob shape do not match INFO:model_loader:found comp_7_conv_1_w but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_s but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_b but blob shape do not match INFO:model_loader:found comp_7_conv_2_middle_w but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_middle_s but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_middle_b but blob shape do not match INFO:model_loader:found comp_7_conv_2_w but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_s but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_b but blob shape do not match INFO:model_loader:last_out_L5_w not found INFO:model_loader:last_out_L5_b not found INFO:model_loader:copying conv1_middle_spatbn_relu_rm to gpu_0/conv1_middle_spatbn_relu_rm INFO:model_loader:copying conv1_middle_spatbn_relu_riv to gpu_0/conv1_middle_spatbn_relu_riv INFO:model_loader:copying conv1_spatbn_relu_rm to gpu_0/conv1_spatbn_relu_rm INFO:model_loader:copying conv1_spatbn_relu_riv to gpu_0/conv1_spatbn_relu_riv INFO:model_loader:copying comp_0_spatbn_1_middle_rm to gpu_0/comp_0_spatbn_1_middle_rm INFO:model_loader:copying comp_0_spatbn_1_middle_riv to gpu_0/comp_0_spatbn_1_middle_riv INFO:model_loader:copying comp_0_spatbn_1_rm to gpu_0/comp_0_spatbn_1_rm INFO:model_loader:copying comp_0_spatbn_1_riv to gpu_0/comp_0_spatbn_1_riv INFO:model_loader:copying comp_0_spatbn_2_middle_rm to gpu_0/comp_0_spatbn_2_middle_rm INFO:model_loader:copying comp_0_spatbn_2_middle_riv to gpu_0/comp_0_spatbn_2_middle_riv INFO:model_loader:copying comp_0_spatbn_2_rm to gpu_0/comp_0_spatbn_2_rm INFO:model_loader:copying comp_0_spatbn_2_riv to gpu_0/comp_0_spatbn_2_riv INFO:model_loader:copying comp_1_spatbn_1_middle_rm to gpu_0/comp_1_spatbn_1_middle_rm INFO:model_loader:copying comp_1_spatbn_1_middle_riv to gpu_0/comp_1_spatbn_1_middle_riv INFO:model_loader:copying comp_1_spatbn_1_rm to gpu_0/comp_1_spatbn_1_rm INFO:model_loader:copying comp_1_spatbn_1_riv to gpu_0/comp_1_spatbn_1_riv INFO:model_loader:copying comp_1_spatbn_2_middle_rm to gpu_0/comp_1_spatbn_2_middle_rm INFO:model_loader:copying comp_1_spatbn_2_middle_riv to gpu_0/comp_1_spatbn_2_middle_riv INFO:model_loader:copying comp_1_spatbn_2_rm to gpu_0/comp_1_spatbn_2_rm INFO:model_loader:copying comp_1_spatbn_2_riv to gpu_0/comp_1_spatbn_2_riv INFO:model_loader:found comp_2_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_rm but blob shape do not match INFO:model_loader:found comp_2_spatbn_1_riv but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_middle_rm but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_middle_riv but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_rm but blob shape do not match INFO:model_loader:found comp_2_spatbn_2_riv but blob shape do not match INFO:model_loader:shortcut_projection_2_spatbn_rm not found INFO:model_loader:shortcut_projection_2_spatbn_riv not found INFO:model_loader:found comp_3_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_3_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:copying comp_3_spatbn_1_rm to gpu_0/comp_3_spatbn_1_rm INFO:model_loader:copying comp_3_spatbn_1_riv to gpu_0/comp_3_spatbn_1_riv INFO:model_loader:copying comp_3_spatbn_2_middle_rm to gpu_0/comp_3_spatbn_2_middle_rm INFO:model_loader:copying comp_3_spatbn_2_middle_riv to gpu_0/comp_3_spatbn_2_middle_riv INFO:model_loader:copying comp_3_spatbn_2_rm to gpu_0/comp_3_spatbn_2_rm INFO:model_loader:copying comp_3_spatbn_2_riv to gpu_0/comp_3_spatbn_2_riv INFO:model_loader:found comp_4_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_rm but blob shape do not match INFO:model_loader:found comp_4_spatbn_1_riv but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_middle_rm but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_middle_riv but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_rm but blob shape do not match INFO:model_loader:found comp_4_spatbn_2_riv but blob shape do not match INFO:model_loader:shortcut_projection_4_spatbn_rm not found INFO:model_loader:shortcut_projection_4_spatbn_riv not found INFO:model_loader:found comp_5_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_rm but blob shape do not match INFO:model_loader:found comp_5_spatbn_1_riv but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_middle_rm but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_middle_riv but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_rm but blob shape do not match INFO:model_loader:found comp_5_spatbn_2_riv but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_rm but blob shape do not match INFO:model_loader:found comp_6_spatbn_1_riv but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_middle_rm but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_middle_riv but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_rm but blob shape do not match INFO:model_loader:found comp_6_spatbn_2_riv but blob shape do not match INFO:model_loader:shortcut_projection_6_spatbn_rm not found INFO:model_loader:shortcut_projection_6_spatbn_riv not found INFO:model_loader:found comp_7_spatbn_1_middle_rm but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_middle_riv but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_rm but blob shape do not match INFO:model_loader:found comp_7_spatbn_1_riv but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_middle_rm but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_middle_riv but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_rm but blob shape do not match INFO:model_loader:found comp_7_spatbn_2_riv but blob shape do not match INFO:data_parallel_model:Creating checkpoint synchronization net INFO:data_parallel_model:Run checkpoint net INFO:train_net:Starting epoch 0/8 [swscaler @ 0x7d1aa4175d20] Warning: data is not aligned! This can lead to a speed loss [E net_async_base.cc:382] [enforce fail at conv_op_cudnn.cc:520] filter.dim() >= 3 && filter.dim() <= 5. Error from operator: input: "gpu_0/comp_1_sum_2" input: "gpu_0/comp_2_conv_1_middle_w" output: "gpu_0/comp_2_conv_1_middle" name: "" type: "Conv" arg { name: "kernels" ints: 1 ints: 3 ints: 3 } arg { name: "ws_nbytes_limit" i: 67108864 } arg { name: "exhaustive_search" i: 1 } arg { name: "strides" ints: 1 ints: 2 ints: 2 } arg { name: "pads" ints: 0 ints: 1 ints: 1 ints: 0 ints: 1 ints: 1 } arg { name: "order" s: "NCHW" } device_option { device_type: 1 device_id: 0 } engine: "CUDNN"frame #0: c10::ThrowEnforceNotMet(char const, int, char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void const) + 0x78 (0x7f1bb3a76178 in /usr/local/lib/libc10.so)
frame #1: + 0x3b11ba5 (0x7f1b780abba5 in /usr/local/lib/libtorch.so)
frame #2: + 0x3b00e08 (0x7f1b7809ae08 in /usr/local/lib/libtorch.so)
frame #3: + 0x3a7df9d (0x7f1b78017f9d in /usr/local/lib/libtorch.so)
frame #4: caffe2::AsyncNetBase::run(int, int) + 0x118 (0x7f1b768d2588 in /usr/local/lib/libtorch.so)
frame #5: + 0x233ee2a (0x7f1b768d8e2a in /usr/local/lib/libtorch.so)
frame #6: c10::ThreadPool::main_loop(unsigned long) + 0x2b3 (0x7f1bb3a704a3 in /usr/local/lib/libc10.so)
frame #7: + 0xbd66f (0x7f1b8c2bd66f in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x76db (0x7f1bb837b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f1bb86b488f in /lib/x86_64-linux-gnu/libc.so.6)
, op Conv
[E net_async_base.cc:134] Rethrowing exception from the run of 'r2plus1d_train'
WARNING:caffe2.python.workspace:Original python traceback for operator
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 496, in main
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 280, in Train
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/data_parallel_model.py", line 39, in Parallelize_GPU
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/data_parallel_model.py", line 237, in Parallelize
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 217, in create_model_ops
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/model_builder.py", line 128, in build_model
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/r3d_model.py", line 99, in create_model
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/r3d_model.py", line 192, in create_r3d
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/video_model.py", line 249, in add_simple_block
WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/models/video_model.py", line 67, in add_conv
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/cnn.py", line 86, in ConvNd
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/brew.py", line 108, in scope_wrapper
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/helpers/conv.py", line 164, in conv_nd
WARNING:caffe2.python.workspace: File "/root/pytorch/build/caffe2/python/helpers/conv.py", line 123, in _ConvBase
Traceback (most recent call last):
File "/usr/VMZ-master/tools/train_net.py", line 501, in
main()
File "/usr/VMZ-master/tools/train_net.py", line 496, in main
Train(args)
File "/usr/VMZ-master/tools/train_net.py", line 388, in Train
explog
File "/usr/VMZ-master/tools/train_net.py", line 123, in RunEpoch
workspace.RunNet(train_model.net.Proto().name)
File "/root/pytorch/build/caffe2/python/workspace.py", line 254, in RunNet
StringifyNetName(name), num_iter, allow_fail,
File "/root/pytorch/build/caffe2/python/workspace.py", line 215, in CallWithExceptionIntercept
return func( args, *kwargs)
RuntimeError: [enforce fail at conv_op_cudnn.cc:520] filter.dim() >= 3 && filter.dim() <= 5.
Error from operator:
input: "gpu_0/comp_1_sum_2" input: "gpu_0/comp_2_conv_1_middle_w" output: "gpu_0/comp_2_conv_1_middle" name: "" type: "Conv" arg { name: "kernels" ints: 1 ints: 3 ints: 3 } arg { name: "ws_nbytes_limit" i: 67108864 } arg { name: "exhaustive_search" i: 1 } arg { name: "strides" ints: 1 ints: 2 ints: 2 } arg { name: "pads" ints: 0 ints: 1 ints: 1 ints: 0 ints: 1 ints: 1 } arg { name: "order" s: "NCHW" } device_option { device_type: 1 device_id: 0 } engine: "CUDNN"frame #0: c10::ThrowEnforceNotMet(char const, int, char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void const ) + 0x78 (0x7f1bb3a76178 in /usr/local/lib/libc10.so)
frame #1: + 0x3b11ba5 (0x7f1b780abba5 in /usr/local/lib/libtorch.so)
frame #2: + 0x3b00e08 (0x7f1b7809ae08 in /usr/local/lib/libtorch.so)
frame #3: + 0x3a7df9d (0x7f1b78017f9d in /usr/local/lib/libtorch.so)
frame #4: caffe2::AsyncNetBase::run(int, int) + 0x118 (0x7f1b768d2588 in /usr/local/lib/libtorch.so)
frame #5: + 0x233ee2a (0x7f1b768d8e2a in /usr/local/lib/libtorch.so)
frame #6: c10::ThreadPool::main_loop(unsigned long) + 0x2b3 (0x7f1bb3a704a3 in /usr/local/lib/libc10.so)
frame #7: + 0xbd66f (0x7f1b8c2bd66f in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x76db (0x7f1bb837b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f1bb86b488f in /lib/x86_64-linux-gnu/libc.so.6)
34
in networkr2plus1d_train
in exception above (most recent call last): WARNING:caffe2.python.workspace: File "/usr/VMZ-master/tools/train_net.py", line 501, in