Error while extracting features

[abhilashi]

WARNING:caffe2.python.workspace:Original python traceback for operator 1 in network Extract Features_init in exception above (most recent call last):
WARNING:caffe2.python.workspace:  File "tools/extract_features.py", line 292, in <module>
WARNING:caffe2.python.workspace:  File "tools/extract_features.py", line 287, in main
WARNING:caffe2.python.workspace:  File "tools/extract_features.py", line 118, in ExtractFeatures
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/data_parallel_model.py", line 32, in Parallelize_GPU
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/data_parallel_model.py", line 209, in Parallelize
WARNING:caffe2.python.workspace:  File "tools/extract_features.py", line 110, in create_model_ops
WARNING:caffe2.python.workspace:  File "/home/ubuntu/dedup/R2Plus1D/lib/models/model_builder.py", line 128, in build_model
WARNING:caffe2.python.workspace:  File "/home/ubuntu/dedup/R2Plus1D/lib/models/r3d_model.py", line 99, in create_model
WARNING:caffe2.python.workspace:  File "/home/ubuntu/dedup/R2Plus1D/lib/models/r3d_model.py", line 145, in create_r3d
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/cnn.py", line 86, in ConvNd
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/brew.py", line 107, in scope_wrapper
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/helpers/conv.py", line 164, in conv_nd
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/helpers/conv.py", line 88, in _ConvBase
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/model_helper.py", line 216, in create_param
WARNING:caffe2.python.workspace:  File "/home/ubuntu/pytorch/build/caffe2/python/modeling/initializers.py", line 30, in create_param
Traceback (most recent call last):
  File "tools/extract_features.py", line 292, in <module>
    main()
  File "tools/extract_features.py", line 287, in main
    ExtractFeatures(args)
  File "tools/extract_features.py", line 121, in ExtractFeatures
    workspace.RunNetOnce(model.param_init_net)
  File "/home/ubuntu/pytorch/build/caffe2/python/workspace.py", line 201, in RunNetOnce
    StringifyProto(net),
  File "/home/ubuntu/pytorch/build/caffe2/python/workspace.py", line 180, in CallWithExceptionIntercept
    return func(*args, **kwargs)
RuntimeError: [enforce fail at operator.cc:209] op. Cannot create operator of type 'MSRAFill' on the device 'CUDA'. Verify that implementation for the corresponding device exist. It might also happen if the binary is not linked with the operator implementation code. If Python frontend is used it might happen if dyndep.InitOpsLibrary call is missing. Operator def: output: "gpu_0/conv1_middle_w" name: "" type: "MSRAFill" arg { name: "shape" ints: 45 ints: 3 ints: 1 ints: 7 ints: 7 } device_option { device_type: 1 cuda_gpu_id: 0 }

I'm on Ubuntu 16.04. This happens when I run the following:

python tools/extract_features.py --test_data=../dups_lmdb_data --model_name=r2plus1d --model_depth=34 --clip_length_rgb=32 --batch_size=4 --load_model_path=./trained/r2.5d_d34_l32_ft_sports1m.pkl --output_path=dups_features.pkl --features=softmax,final_avg,video_id --sanity_check=0 --get_video_id=1 --use_local_file=1 --num_labels=400

I'm on a AWS general purpose compute machine (doesn't have GPU).

[abhilashi]

@dutran appreciate any leads/tips to debug this :)

[dutran]

@abhilashi Currently feature_extraction tool need to run on GPU and it does not support CPU only mode.

[abhilashi]

@dutran thanks for the quick reply. Trying GPU now 👍