Open austinmw opened 6 years ago
Maybe changing the sync_replicas
or replicas_to_aggregate
parameters in your configuration might change something. See https://stackoverflow.com/a/51724431/448357
Hi , Struck with the same issue ..... update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS, first_clone_scope) training stops at this line Did u find any solution ?? working fine for other models
I've previously tried two models:
ssd_mobilenet_v1_coco_2017_11_17
andfaster_rcnn_inception_resnet_v2_atrous_coco_2018_01_28
which will both run like this:run_train() { export CUDA_VISIBLE_DEVICES=0,1,2 python3 /home//training/train.py --logtostderr --pipeline_config_path=/home//training/faster_rcnn_nas_coco.config --train_dir=/home//training/models/train --num_clones=3 --ps_tasks=1
unset CUDA_VISIBLE_DEVICES
}
I have 4 GPU's so I've been setting the first three to train and the last one to eval. However, for some reason I'm unable to do the same for the model
faster_rcnn_nas_coco_2018_01_28
. I get the error:Could anyone please explain why this is or how I can fix it?