facebookresearch / Detectron

FAIR's research platform for object detection research, implementing popular algorithms like Mask R-CNN and RetinaNet.
Apache License 2.0
26.22k stars 5.45k forks source link

Trying to use my customized training weight, but error occurs.. #931

Open seokhyeonSong opened 5 years ago

seokhyeonSong commented 5 years ago

PLEASE FOLLOW THESE INSTRUCTIONS BEFORE POSTING

  1. Please thoroughly read README.md, INSTALL.md, GETTING_STARTED.md, and FAQ.md
  2. Please search existing open and closed issues in case your issue has already been reported
  3. Please try to debug the issue in case you can solve it on your own before posting

After following steps 1-3 above and agreeing to provide the detailed information requested below, you may continue with posting your issue

Expected results

What did you expect to see?

Actual results

What did you observe instead?

Detailed steps to reproduce

E.g.: I trained with my dataset and ran infer_simple.py but error occurs..

Traceback (most recent call last):
  File "tools/infer_simple.py", line 185, in <module>
    main(args)
  File "tools/infer_simple.py", line 153, in main
    model, im, None, timers=timers
  File "/home/git/DeepFashion2/Detectron/detectron/core/test.py", line 66, in im_detect_all
    model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, boxes=box_proposals
  File "/home/git/DeepFashion2/Detectron/detectron/core/test.py", line 158, in im_detect_bbox
    workspace.RunNet(model.net.Proto().name)
  File "/home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/workspace.py", line 254, in RunNet
    StringifyNetName(name), num_iter, allow_fail,
  File "/home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/workspace.py", line 215, in CallWithExceptionIntercept
    return func(*args, **kwargs)
RuntimeError: IsType<T>() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch-nightly_1565028713490/work/aten/src/ATen/core/blob.h:77, please report a bug to PyTorch. wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor.
Offending Blob name: gpu_0/res4_6_branch2a_w.
Error from operator: 
input: "gpu_0/res4_5_branch2c_bn" input: "gpu_0/res4_6_branch2a_w" output: "gpu_0/res4_6_branch2a" name: "" type: "Conv" arg { name: "kernel" i: 1 } arg { name: "order" s: "NCHW" } arg { name: "stride" i: 1 } arg { name: "pad" i: 0 } arg { name: "dilation" i: 1 } arg { name: "exhaustive_search" i: 0 } device_option { device_type: 1 device_id: 0 } engine: "CUDNN" (Get at /opt/conda/conda-bld/pytorch-nightly_1565028713490/work/aten/src/ATen/core/blob.h:77)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f4c04b7ae37 in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: caffe2::Tensor const& caffe2::Blob::Get<caffe2::Tensor>() const + 0x18c (0x7f4c33f792ec in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-37m-x86_64-linux-gnu.so)
frame #2: caffe2::Tensor const& caffe2::OperatorBase::Input<caffe2::Tensor>(int, c10::DeviceType) + 0x3ea (0x7f4c340149da in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-37m-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x3cc40a8 (0x7f4c08c8d0a8 in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libtorch.so)
frame #4: <unknown function> + 0x3cb18b0 (0x7f4c08c7a8b0 in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libtorch.so)
frame #5: <unknown function> + 0x3c38ed5 (0x7f4c08c01ed5 in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libtorch.so)
frame #6: caffe2::AsyncNetBase::run(int, int) + 0x10e (0x7f4c07506d3e in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libtorch.so)
frame #7: <unknown function> + 0x254422a (0x7f4c0750d22a in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libtorch.so)
frame #8: c10::ThreadPool::main_loop(unsigned long) + 0x293 (0x7f4c04b76543 in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #9: <unknown function> + 0x38c8f (0x7f4c04b7fc8f in /home/git/anaconda2/envs/arc2/lib/python3.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #10: <unknown function> + 0x76db (0x7f4c4c8ec6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #11: clone + 0x3f (0x7f4c4c61588f in /lib/x86_64-linux-gnu/libc.so.6)

The command that you ran



### System information

* Operating system: ubuntu 16.04
* Compiler version: GCC 5.4.0
* CUDA version: 10.1
* cuDNN version: 7.x
* NVIDIA driver version: 418.67
* GPU models (for all devices if they are not all the same): 2080ti
* `python --version` output: python 3.7.3