search

youtubevos / MaskTrackRCNN

MaskTrackRCNN for video instance segmentation based on mmdetection
Apache License 2.0
431 stars 75 forks source link

RuntimeError #18

Closed ruyueshuo closed 4 years ago

ruyueshuo commented 4 years ago

Hi, I'm train MaskTrackRCNN following the readme,. However, an error occured.

Traceback (most recent call last): File "/home/ubuntu/.pycharm_helpers/pydev/pydevd.py", line 2060, in main() File "/home/ubuntu/.pycharm_helpers/pydev/pydevd.py", line 2054, in main globals = debugger.run(setup['file'], None, None, is_module) File "/home/ubuntu/.pycharm_helpers/pydev/pydevd.py", line 1405, in run return self._exec(is_module, entry_point_fn, module_name, file, globals, locals) File "/home/ubuntu/.pycharm_helpers/pydev/pydevd.py", line 1412, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/ubuntu/code/MaskTrackRCNN/tools/train.py", line 99, in main() File "/home/ubuntu/code/MaskTrackRCNN/tools/train.py", line 92, in main logger=logger) File "/home/ubuntu/code/MaskTrackRCNN/mmdet/apis/train.py", line 59, in train_detector _non_dist_train(model, dataset, cfg, validate=validate) File "/home/ubuntu/code/MaskTrackRCNN/mmdet/apis/train.py", line 123, in _non_dist_train runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/mmcv/runner/runner.py", line 358, in run epoch_runner(data_loaders[i], **kwargs) File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/mmcv/runner/runner.py", line 271, in train self.call_hook('after_train_iter') File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/mmcv/runner/runner.py", line 229, in call_hook getattr(hook, fn_name)(self) File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 17, in after_train_iter runner.outputs['loss'].backward() File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/ubuntu/Softwares/anaconda3/envs/MaskTrackRCNN/lib/python3.7/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [148, 256, 14, 14]], which is output 0 of IndexPutBackward, is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

with: cuda:9.2 pytorch:1.2.0

Could you please give me any suggestion?

ruyueshuo commented 4 years ago

Problem has been solved by setting pytorch version to 1.1.0.