torch error when loading LayoutParserPredictor: object has no attribute nms

[e-tornike]

Hi there,

thanks for the cool project! :sunglasses:

Describe the bug I'm running into a tricky [torchvision version?] effdet/jit error when loading LayoutParserPredictor (running the vila_for_scidoc_parsing example from mmda).

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version, see the Layout Parser Releases

To Reproduce Steps to reproduce the behavior:

1. Running the main.py script found here with the following arguments:

   python main.py 
   --pdf-path ./FILE.pdf 
   --vila-type hvila 
   --vila-model-path allenai/hvila-block-layoutlm-finetuned-docbank 
   --export-folder ./tmp/

Environment

1. Ubuntu 20.04
2. Python 3.8.10
3. Installed everything via pip:

   effdet==0.3.0
   layoutparser==0.3.4
   mmda @ git+https://github.com/allenai/mmda.git@e644f3cec4041b57dbcd3ba2b8bc10eadb71ac09
   torch==1.11.0+cu113
   torchvision==0.12.0+cu113
   transformers==4.5.0
   vila==0.2.1

Error traceback

Traceback (most recent call last):
  File "main.py", line 126, in <module>
    layout_predictor = LayoutParserPredictor.from_pretrained(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/mmda/predictors/lp_predictors.py", line 36, in from_pretrained
    model = lp.AutoLayoutModel(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/layoutparser/file_utils.py", line 223, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/layoutparser/file_utils.py", line 232, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/layoutparser/models/__init__.py", line 17, in <module>
    from .effdet.layoutmodel import EfficientDetLayoutModel
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/layoutparser/models/effdet/__init__.py", line 16, in <module>
    from .layoutmodel import EfficientDetLayoutModel
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/layoutparser/models/effdet/layoutmodel.py", line 28, in <module>
    from effdet import create_model
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/effdet/__init__.py", line 2, in <module>
    from .bench import DetBenchPredict, DetBenchTrain, unwrap_bench
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/effdet/bench.py", line 70, in <module>
    def _batch_detection(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 1318, in script
    fn = torch._C._jit_script_compile(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_recursive.py", line 841, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 1318, in script
    fn = torch._C._jit_script_compile(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_recursive.py", line 841, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 1318, in script
    fn = torch._C._jit_script_compile(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_recursive.py", line 841, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 1318, in script
    fn = torch._C._jit_script_compile(
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_recursive.py", line 841, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 1318, in script
    fn = torch._C._jit_script_compile(
RuntimeError: 
object has no attribute nms:
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 40
        _log_api_usage_once(nms)
    _assert_has_ops()
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
           ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
'nms' is being compiled since it was called from '_batched_nms_vanilla'
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 108
    for class_id in torch.unique(idxs):
        curr_indices = torch.where(idxs == class_id)[0]
        curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        keep_mask[curr_indices[curr_keep_indices]] = True
    keep_indices = torch.where(keep_mask)[0]
'_batched_nms_vanilla' is being compiled since it was called from 'batched_nms'
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 72
    # https://github.com/pytorch/vision/issues/1311#issuecomment-781329339
    if boxes.numel() > (4000 if boxes.device.type == "cpu" else 20000) and not torchvision._is_tracing():
        return _batched_nms_vanilla(boxes, scores, idxs, iou_threshold)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    else:
        return _batched_nms_coordinate_trick(boxes, scores, idxs, iou_threshold)
'batched_nms' is being compiled since it was called from 'generate_detections'
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/effdet/anchors.py", line 140
        scores[top_detection_idx] = soft_scores
    else:
        top_detection_idx = batched_nms(boxes, scores, classes, iou_threshold=0.5)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

    # keep only top max_det_per_image scoring predictions
'generate_detections' is being compiled since it was called from '_batch_detection'
  File "/home/user/mmda_example/venv/lib/python3.8/site-packages/effdet/bench.py", line 82
        img_scale_i = None if img_scale is None else img_scale[i]
        img_size_i = None if img_size is None else img_size[i]
        detections = generate_detections(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            class_out[i], box_out[i], anchor_boxes, indices[i], classes[i],
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            img_scale_i, img_size_i, max_det_per_image=max_det_per_image, soft_nms=soft_nms)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        batch_detections.append(detections)
    return torch.stack(batch_detections, dim=0)

Additional context Would appreciate any tips for resolving this. It appears related to this issue, where simply providing the torchvision version seems to have solved it for some users.

I apologize if I am posting this on the wrong repository. If needed, I can repost it to mmda.

Thanks in advance! :pray:

[lolipopshock]

Thank you @e-tornike for reporting this issue!

I feel a straightforward solution is to just create another environment and re-install torch (even with a CPU version), and test it it would work. Also you might want to try an earlier PyTorch version (e.g., 1.10 or 1.9) along with other decencies. I do see PyTorch 1.11 is released in March 2022 yet these models are trained on year ago; we haven't thorough tested the models on newer version, and maybe that's the issue. Let me know if it can be fixed. Thanks!