Closed rgualan closed 6 years ago
You can train 3 models on single GPU. Just don't use --multi_gpu flag for trainer.py. You can also reduce memory requirements for training using --eval_memsize parameter (default is 5, try 2 or 1) and for prediction using batch_size parameter in predict()
Single model and multiple model graphs have slightly different structure, your error seems like you trained 3-models graph and trying to use it in single-model regime.
First of all, thanks for the excellent code. Now the problem: Since I only have one GPU (Nvidia Quadro), I was able to run only one model by means of:
python trainer.py --name s32 --hparam_set=s32 --n_models=1 --name s32 --no_eval --no_forward_split --asgd_decay=0.99 --max_steps=11500 --save_from_step=10500
When I try to execute the submission-final file, I changed the corresponding cell as follows:
for tm in range(1): tf.reset_default_graph() t_preds.append(predict(paths, build_hparams(hparams.params_s32), back_offset=0, predict_window=63, n_models=1, target_model=tm, seed=2, batch_size=2048, asgd=True))
to account for only one model's checkpoints. However, I am getting an error that I cannot solve:
tf.reset_default_graph()
preds = predict(paths, default_hparams(), back_offset=0,
n_models=3, target_model=0, seed=2, batch_size=2048, asgd=True)
t_preds = []
for tm in range(1):
ValueError Traceback (most recent call last)