search

dbolya / yolact

A simple, fully convolutional model for real-time instance segmentation.
MIT License
4.98k stars 1.32k forks source link

Training crashes after a number of epochs #760

Open mirceamp opened 2 years ago

mirceamp commented 2 years ago

I am training yolact on a custom dataset. However after some epochs the training crashes. I tried to restart the training from the last weights correctly computed and for some time it continued training and then it crashes again. Now I am at epoch 58 and if I try to restart the training it crashes. So at this point it doesn't continue. The error which is displayed when it crashes is shown bellow.

Traceback (most recent call last): File "/home/mmp/Downloads/yolact/train.py", line 504, in train() File "/home/mmp/Downloads/yolact/train.py", line 371, in train compute_validation_map(epoch, iteration, yolact_net, val_dataset, log if args.log else None) File "/home/mmp/Downloads/yolact/train.py", line 492, in compute_validation_map val_info = eval_script.evaluate(yolact_net, dataset, train_mode=True) File "/home/mmp/Downloads/yolact/eval.py", line 964, in evaluate prep_metrics(ap_data, preds, img, gt, gt_masks, h, w, num_crowd, dataset.ids[image_idx], detections) File "/home/mmp/Downloads/yolact/eval.py", line 403, in prep_metrics classes, scores, boxes, masks = postprocess(dets, w, h, crop_masks=args.crop, score_threshold=args.score_threshold) File "/home/mmp/Downloads/yolact/layers/output_utils.py", line 39, in postprocess print(dets['class'].size()) TypeError: 'NoneType' object is not subscriptable

Does anyone know how to solve this issue?

qlianghe commented 5 months ago

I had the same problem, have you solved it yet?