ZENGXH
commented
5 years ago It looks like the training dataloader is loading a wrong proposal file (i.e. the proposal file it loaded is for eval instead of training: experiments/proposals/coco81/inference/youtubevos_val200_meta/asdict_50/pred_DICT.pth).
Could you check the flag pred_offline_path? It should point to something like experiments/proposals/coco81/inference/youtubevos_train3k_meta/asdict_50/pred_DICT.pth.
You may also want to check the scripts/train/*sh file.
Coolxiaolin
when i run python train.py, the following error occurs:
2019-10-16 09:48:22,325-{train.py:384}-INFO-[model_name] ytb_r50_w11 2019-10-16 09:48:22,325-{train.py:385}-INFO-get number of gpu: 1 2019-10-16 09:48:23,989-{utils.py:213}-INFO-[load_DMM_config] dmm/configs/train.yaml 2019-10-16 09:48:23,996-{utils.py:232}-INFO-ud relax_max_iter 400 -> 10|ud relax_proj_iter 50 -> 5 2019-10-16 09:48:24,041-{utils.py:213}-INFO-[load_DMM_config] dmm/configs/train.yaml 2019-10-16 09:48:24,047-{utils.py:232}-INFO-ud relax_max_iter 400 -> 10|ud relax_proj_iter 50 -> 5 2019-10-16 09:48:24,050-{train.py:152}-INFO-{'sort_max_num': 50, 'matching_score_thre': 0.0, 'score_weight': 0.3, 'relax': 1, 'relax_max_iter': 10, 'relax_proj_iter': 5, 'relax_topk': 0, 'relax_learning_rate': 0.1, 'matching': {'match_max_score': 1, 'algo': 'relax', 'cost': 'cosine'}, 'encoder': {'nms_thresh': 0.4}} 2019-10-16 09:48:26,367-{trainer.py:63}-INFO-load from json_data; num vid 3000 2019-10-16 09:48:26,367-{train.py:154}-INFO-init model 4.042 2019-10-16 09:48:26,369-{train.py:161}-INFO-optimizer 0.001 2019-10-16 09:48:26,369-{train.py:163}-INFO-[enc_opt] len: 2; len for each param group: [48, 161] 2019-10-16 09:48:26,369-{train.py:165}-INFO-[dec_opt] len: 1; len for each param group: [10] 2019-10-16 09:48:26,371-{train.py:213}-INFO-save args in experiments/ytb_r50_w11/10-16-09-48args.pkl 2019-10-16 09:48:26,371-{train.py:214}-INFO-Namespace(augment=False, base_model='resnet50', batch_size=4, best_val_loss=0, cache_data=1, config_train='dmm/configs/train.yaml', dataset='youtube', davis_eval_folder='', device=device(type='cuda', index=0), distributed=0, distributed_manully=0, distributed_manully_Nrep=0, distributed_manully_rank=0, dropout=0.0, epoch_resume=0, eval_flag='pred', eval_split='trainval', finetune_after=0, gpu_id=0, gt_maxseqlen=5, hidden_size=128, imsize=480, iou_weight=1.0, kernel_size=3, length_clip=3, load_proposals=1, load_proposals_dataset=1, local_rank=0, log_file='train.log', log_term=False, loss_weight_iouraw=18.0, loss_weight_match=1.0, lr=0.001, lr_cnn=0.0001, lr_decoder=0.001, mask_th=0.5, max_dets=100, max_epoch=100, max_eval_iter=800, maxseqlen=5, min_delta=0.0, min_size=0.001, model_dir='experiments/ytb_r50_w11', model_name='ytb_r50_w11', models_root='experiments/', momentum=0.9, my_augment=False, ngpus=1, num_classes=21, num_workers=4, only_spatial=False, only_temporal=False, optim='adam', optim_cnn='adam', overwrite_loadargs=1, pad_video=0, patience=15, patience_stop=60, pred_offline_meta='data/ytb_vos/splits_813_3k_trainvaltest/meta_vid_frame_2_predid.json', pred_offline_path=['experiments/proposals/coco81/inference/youtubevos_val200_meta/asdict_50/pred_DICT.pth'], pred_offline_path_eval=None, prev_mask_d=1, print_every=2, random_select_frames=0, resize=False, resume=False, resume_path='epoxx_iterxxxx', rotation=10, sample_inference_mask=0, save_every=3000, seed=123, shear=0.1, single_object=False, skip_empty_starting_frame=0, skip_mode='concat', test=0, test_image_h=256, test_image_w=448, test_model_path='', threshold_mask=0.4, train_h=255, train_split='train', train_w=448, translation=0.1, update_encoder=1, use_gpu=True, use_refmask=0, weight_decay=1e-06, weight_decay_cnn=1e-06, year='2017', youtube_dir='../../databases/YouTubeVOS/', zoom=0.7) 2019-10-16 09:48:26,372-{train.py:223}-INFO-init_dataloaders 2019-10-16 09:48:26,412-{dataset.py:119}-INFO-[train] loading offline from experiments/proposals/coco81/inference/youtubevos_val200_meta/asdict_50/pred_DICT.pth; Nf ['experiments/proposals/coco81/inference/youtubevos_val200_meta/asdict_50/pred_DICT.pth'] 2019-10-16 09:48:27,298-{dataset.py:125}-INFO-+new_parts 200: 0.8864507675170898 2019-10-16 09:48:27,303-{dataset.py:133}-INFO-load offline use 0.89 | len 200 2019-10-16 09:48:27,320-{youtubeVOS.py:84}-INFO-[dataset] phase read train; len of db seq 3000 2019-10-16 09:48:27,320-{youtubeVOS.py:103}-INFO-LMDB not found. This could affect the data loading time. It is recommended to use LMDB. 2019-10-16 09:48:27,321-{youtubeVOS.py:115}-INFO-no cache data found at data/ytb_vos/splits_813_3k_trainvaltest/dmm_cached_train.pkl; it will take a while to cache the data 2019-10-16 10:17:41,177-{youtubeVOS.py:121}-INFO-try to dump in data/ytb_vos/splits_813_3k_trainvaltest/dmm_cached_train.pkl 2019-10-16 10:18:03,726-{youtubeVOS.py:125}-INFO-load lmdb 1776.42 Traceback (most recent call last): File "/home/zhanglin/Research/codes/2020/DMM_Net/train.py", line 403, in
trainIters(args)
File "/home/zhanglin/Research/codes/2020/DMM_Net/train.py", line 225, in trainIters
loaders = init_dataloaders(args)
File "/home/zhanglin/Research/codes/2020/DMM_Net/train.py", line 86, in init_dataloaders
use_prev_mask = False)
File "/home/zhanglin/Research/codes/2020/DMM_Net/dmm/dataloader/dataset_utils.py", line 17, in get_dataset
use_prev_mask = use_prev_mask)
File "/home/zhanglin/Research/codes/2020/DMM_Net/dmm/dataloader/youtubeVOS.py", line 157, in init
images_valid = [fname for img, fname in zip(images, seq.files) if self.countobj[dbname][img] > 0 ]
File "/home/zhanglin/Research/codes/2020/DMM_Net/dmm/dataloader/youtubeVOS.py", line 157, in
images_valid = [fname for img, fname in zip(images, seq.files) if self.countobj[dbname][img] > 0 ]
KeyError: '003234408d'
Process finished with exit code 1
i don't know why....looking forward to your reply