Check failed: param_.op_type != "" ( vs. ) Custom operator type missing

[weiliuxm]

Hi, Guo! I ran the script vgg_voc07.sh, and encountering the following mistake:

INFO:root:########## GENERATE RPN DETECTION {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [8, 16, 32], 'FIXED_PARAMS': ['conv1', 'conv2'], 'FIXED_PARAMS_SHARED': ['conv1', 'conv2', 'conv3', 'conv4', 'conv5'], 'IMAGE_STRIDE': 0, 'NUM_ANCHORS': 9, 'NUM_CLASSES': 21, 'PIXEL_MEANS': array([ 103.939, 116.779, 123.68 ]), 'RCNN_FEAT_STRIDE': 16, 'RPN_FEAT_STRIDE': 16, 'SCALES': [(600, 1000)], 'TEST': {'BATCH_IMAGES': 1, 'CXX_PROPOSAL': True, 'HAS_RPN': True, 'NMS': 0.3, 'PROPOSAL_MIN_SIZE': 16, 'PROPOSAL_NMS_THRESH': 0.7, 'PROPOSAL_POST_NMS_TOP_N': 2000, 'PROPOSAL_PRE_NMS_TOP_N': 20000, 'RPN_MIN_SIZE': 16, 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000}, 'TRAIN': {'ASPECR_GROUPING': True, 'BATCH_IMAGES': 1, 'BATCH_ROIS': 128, 'BBOX_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZATION_PRECOMPUTED': False, 'BBOX_REGRESSION_THRESH': 0.5, 'BBOX_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_WEIGHTS': array([ 1., 1., 1., 1.]), 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'CXX_PROPOSAL': True, 'END2END': False, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'RPN_BATCH_SIZE': 256, 'RPN_BBOX_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 16, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOPN': 12000}} [22:44:04] /home/weiliu/mxnet/dmlc-core/include/dmlc/./logging.h:300: [22:44:04] src/operator/./custom-inl.h:126: Check failed: param.op_type != "" ( vs. ) Custom operator type missing

Stack trace returned 25 entries: [bt] (0) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fd8720a492c] [bt] (1) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2op12CustomOpProp4InitERKSt6vectorISt4pairISsSsESaIS4_EE+0xaaf) [0x7fd872a4473f] [bt] (2) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2op12ParsedOpProp4InitERKN4nnvm9NodeAttrsE+0xd3) [0x7fd8729aede3] [bt] (3) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(+0xf61f0a) [0x7fd8729a4f0a] [bt] (4) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4nnvm6Symbol13CreateFunctorEPKNS_2OpESt13unordered_mapISsSsSt4hashISsESt8equal_toISsESaISt4pairIKSsSsEEE+0x98) [0x7fd87366f208] [bt] (5) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(MXSymbolCreateAtomicSymbol+0x6a9) [0x7fd8728a91f9] [bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fd8b37c3adc] [bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fd8b37c340c] [bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7fd8b39da5fe] [bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7fd8b39dbf9e] [bt] (10) python(PyEval_EvalFrameEx+0x98d) [0x5244dd] [bt] (11) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (12) python(PyEval_EvalFrameEx+0x1a10) [0x525560] [bt] (13) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (14) python(PyEval_EvalFrameEx+0x7e8) [0x524338] [bt] (15) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (16) python(PyEval_EvalFrameEx+0x1a10) [0x525560] [bt] (17) python(PyEval_EvalFrameEx+0xc9a) [0x5247ea] [bt] (18) python(PyEval_EvalFrameEx+0xc9a) [0x5247ea] [bt] (19) python() [0x567d14] [bt] (20) python(PyRun_FileExFlags+0x92) [0x465bf4] [bt] (21) python(PyRun_SimpleFileExFlags+0x2ee) [0x46612d] [bt] (22) python(Py_Main+0xb5e) [0x466d92] [bt] (23) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd8b4d91f45] [bt] (24) python() [0x577c2e]

Traceback (most recent call last): File "train_alternate.py", line 110, in main() File "train_alternate.py", line 107, in main args.rcnn_epoch, args.rcnn_lr, args.rcnn_lr_step) File "train_alternate.py", line 36, in alternate_train vis=False, shuffle=False, thresh=0) File "/home/weiliu/projects/traffSgn/mx-rcnn/rcnn/tools/test_rpn.py", line 23, in testrpn sym = eval('get' + network + '_rpn_test')(num_anchors=config.NUM_ANCHORS) File "/home/weiliu/projects/traffSgn/mx-rcnn/rcnn/symbol/symbol_vgg.py", line 233, in get_vgg_rpn_test threshold=config.TEST.PROPOSAL_NMS_THRESH, rpn_min_size=config.TEST.PROPOSAL_MIN_SIZE) File "/home/weiliu/mxnet/python/mxnet/_ctypes/symbol.py", line 181, in creator ctypes.byref(sym_handle))) File "/home/weiliu/mxnet/python/mxnet/base.py", line 75, in check_call raise MXNetError(py_str(LIB.MXGetLastError())) mxnet.base.MXNetError: [22:44:04] src/operator/./custom-inl.h:126: Check failed: param.op_type != "" ( vs. ) Custom operator type missing

Stack trace returned 25 entries: [bt] (0) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fd8720a492c] [bt] (1) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2op12CustomOpProp4InitERKSt6vectorISt4pairISsSsESaIS4_EE+0xaaf) [0x7fd872a4473f] [bt] (2) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2op12ParsedOpProp4InitERKN4nnvm9NodeAttrsE+0xd3) [0x7fd8729aede3] [bt] (3) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(+0xf61f0a) [0x7fd8729a4f0a] [bt] (4) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4nnvm6Symbol13CreateFunctorEPKNS_2OpESt13unordered_mapISsSsSt4hashISsESt8equal_toISsESaISt4pairIKSsSsEEE+0x98) [0x7fd87366f208] [bt] (5) /home/weiliu/mxnet/python/mxnet/../../lib/libmxnet.so(MXSymbolCreateAtomicSymbol+0x6a9) [0x7fd8728a91f9] [bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fd8b37c3adc] [bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fd8b37c340c] [bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7fd8b39da5fe] [bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7fd8b39dbf9e] [bt] (10) python(PyEval_EvalFrameEx+0x98d) [0x5244dd] [bt] (11) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (12) python(PyEval_EvalFrameEx+0x1a10) [0x525560] [bt] (13) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (14) python(PyEval_EvalFrameEx+0x7e8) [0x524338] [bt] (15) python(PyEval_EvalCodeEx+0x2b1) [0x555551] [bt] (16) python(PyEval_EvalFrameEx+0x1a10) [0x525560] [bt] (17) python(PyEval_EvalFrameEx+0xc9a) [0x5247ea] [bt] (18) python(PyEval_EvalFrameEx+0xc9a) [0x5247ea] [bt] (19) python() [0x567d14] [bt] (20) python(PyRun_FileExFlags+0x92) [0x465bf4] [bt] (21) python(PyRun_SimpleFileExFlags+0x2ee) [0x46612d] [bt] (22) python(Py_Main+0xb5e) [0x466d92] [bt] (23) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd8b4d91f45] [bt] (24) python() [0x577c2e]

Could you help me, please? Thank you in advance!

[ijkguo]

1. If you use dmlc/mxnet, please try my fork, simple branch (basis v0.9.1). It has additional operators.
2. Then please try mxnet-v0.8 (my fork of mxnet, legacy branch) if you have not done that. I know that v0.8 works with test_rpn.
3. I am verifying results on v0.9.1. If there is a problem, I will fix it once I get there.

[ijkguo]

I cannot reproduce your issue on v0.8 nor v0.9.1. In rcnn/symbol/symbol_vgg.py line 233, in get_vgg_rpn_test threshold=config.TEST.PROPOSAL_NMS_THRESH, rpn_min_size=config.TEST.PROPOSAL_MIN_SIZE) It is not a customop but rather a C++ operator. I cannot see why there would be any error on customop

[weiliuxm]

Thank you for you reply sincerely.

Maybe I have change some lines of you codes, leading to the error. The MXNet installed was from: https://github.com/dmlc/mxnet

Is there any instruction to install your fork of MXNet? I have failed to install it many times. It seems not easy to install as dmlc/mxnet. I am a beginner for Python and MXNet.

Thank you in advance!

[ijkguo]

It is the same. The difference is that I have added some operators but you can still follow http://mxnet.io/get_started/setup.html#overview to install.

[weiliuxm]

Thank you for you advice. I have install your fork of MXNet. The command to download the fork is: git clone https://github.com/precedenceguo/mx-rcnn.git --recursive I missed "recursive" out.

How to set parameter to read checkpoint to run the script vgg_alter_voc07.sh? Every time I debug the code to train RPN, it begins with epoch[0], although I have run 8 epoches in the last time and rpn1-0008.params was saved.

Thank you in advance!

[ijkguo]

e.g. --resume --begin_epoch 8 --end_epoch 16 --lr 0.0001 --lr_epoch 16

[weiliuxm]

I thought it can read checkpoints automaticly. Thank you for your reply. In the stage of generating RPN detection, I encountered the following bug: INFO:root:########## GENERATE RPN DETECTION {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [8, 16, 32], 'FIXED_PARAMS': ['conv1', 'conv2'], 'FIXED_PARAMS_SHARED': ['conv1', 'conv2', 'conv3', 'conv4', 'conv5'], 'IMAGE_STRIDE': 0, 'NUM_ANCHORS': 9, 'NUM_CLASSES': 21, 'PIXEL_MEANS': array([ 103.939, 116.779, 123.68 ]), 'RCNN_FEAT_STRIDE': 16, 'RPN_FEAT_STRIDE': 16, 'SCALES': [(600, 1000)], 'TEST': {'BATCH_IMAGES': 1, 'CXX_PROPOSAL': True, 'HAS_RPN': True, 'NMS': 0.3, 'PROPOSAL_MIN_SIZE': 16, 'PROPOSAL_NMS_THRESH': 0.7, 'PROPOSAL_POST_NMS_TOP_N': 2000, 'PROPOSAL_PRE_NMS_TOP_N': 20000, 'RPN_MIN_SIZE': 16, 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000}, 'TRAIN': {'ASPECR_GROUPING': True, 'BATCH_IMAGES': 1, 'BATCH_ROIS': 128, 'BBOX_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZATION_PRECOMPUTED': False, 'BBOX_REGRESSION_THRESH': 0.5, 'BBOX_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_WEIGHTS': array([ 1., 1., 1., 1.]), 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'CXX_PROPOSAL': True, 'END2END': False, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'RPN_BATCH_SIZE': 256, 'RPN_BBOX_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 16, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 12000}} Traceback (most recent call last): File "train_alternate.py", line 110, in main() File "train_alternate.py", line 107, in main args.rcnn_epoch, args.rcnn_lr, args.rcnn_lr_step) File "train_alternate.py", line 36, in alternate_train vis=False, shuffle=False, thresh=0) File "/home/weiliu/projects/traffSgn/mx-rcnn/rcnn/tools/test_rpn.py", line 23, in testrpn sym = eval('get' + network + '_rpn_test')(num_anchors=config.NUM_ANCHORS) File "/home/weiliu/projects/traffSgn/mx-rcnn/rcnn/symbol/symbol_vgg.py", line 228, in get_vgg_rpn_test group = mx.symbol.Proposal( AttributeError: 'module' object has no attribute 'Proposal'

iFighting encountered the similar error: https://github.com/precedenceguo/mx-rcnn/issues/45 He solved the problem by installing the fork you supply. But, the folk of my MXNet is your. I install the MXNet and mxnet-cnn following the instructions supplied.

[weiliuxm]

Run the following command, encountering the similar error. python train_end2end.py --gpu 0

File "mx-rcnn/rcnn/symbol/symbol_vgg.py", line 359, in get_vgg_train rois = mx.symbol.Proposal( AttributeError: 'module' object has no attribute 'Proposal'

[ijkguo]

Please use pip uninstall mxnet repeatedly until you see no mxnet installed.

echo $PYTHONPATH and see where it is.

[weiliuxm]

And then, what do to? I used sudo pip uninstall mxnet , until "Cannot uninstall requirement mxnet, not installed". Then I reinstalled MXNet and mxnet-cnn, however the error has not been corrected. Thank you sincerely. You are very nice.

[ijkguo]

just copy mxnet/src/operator/proposal-inl.h, proposal.cc proposal.cc to dmlc/mxnet and install it.

[breeze5428]

Thank you for you patience. Now I am able to train and test the VGG model on the VOC dataset.