[track-1] inference 오류

grnd-lab commented 4 years ago

로컬이랑 테스트 submit(nsml submit -t ~~)에서는 오류가 없는데, 실제 제출에서는 inference가 끝난 후에 아래와 같이 오류가 뜹니다. 이유를 알 수 있을까요? infer쪽은 접근이 제한되어 오류 위치나 원인 찾기가 힘드네요.

Building docker image. It may take a while .........load nsml model takes 2.119027853012085 seconds .Infer test set. The inference should be completed within 3600 seconds. .Error occurred while inference. You can check error 'nsml submit --test' TypeError: can only concatenate tuple (not "list") to tuple

Infer test set takes 1138.0046072006226 seconds ..Error: Fail to get prediction result: t0101/tr-1/82/model time="2020/07/30 09:50:15.041" level=fatal msg="Internal server error"

bluebrush commented 4 years ago

@grnd-lab 님, 최근 오류 메시지입니다.

 File "main.py", line 194, in infer
    return _infer(model, root_path)
  File "main.py", line 127, in _infer
    output = mpt(image, output_file=None)
  File "/app/multi_person_tracker/mpt.py", line 194, in __call__
    trackers = self.run_tracker(dataloader)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/app/multi_person_tracker/mpt.py", line 78, in run_tracker
    predictions = self.detector(batch)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/detection/generalized_rcnn.py", line 71, in forward
    detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/detection/roi_heads.py", line 836, in forward
    keypoint_logits = self.keypoint_predictor(keypoint_features)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/detection/keypoint_rcnn.py", line 257, in forward
    x, scale_factor=float(self.up_scale), mode="bilinear", align_corners=False
  File "/opt/conda/lib/python3.6/site-packages/torchvision/ops/misc.py", line 116, in interpolate
    output_shape = input.shape[:-2] + output_shape
TypeError: can only concatenate tuple (not "list") to tuple

grnd-lab commented 4 years ago

감사합니다

AI-GrandChallenge / round-1

[track-1] inference 오류 #102