When i execute the following code:
python trainer_preid.py --output=outputs/ --data=data/market/ --dataset-name=market1501 --batch-size=16 --num-epochs=100 --network-name=resnet_v1_50 --initial-checkpoint=checkpoint/resnet_v1_50.ckpt --checkpoint-exclude-scopes=resnet_v1_50/logits --trainable-scopes=resnet_v1_50/logits --no-evaluation
The error occured:
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.
environment: tensorflow -gpu1.10 python3.65 no matlab(only train model) I comment line 8、11 and 35 in trainer_preid.py, and set do_evaluation=False.
from evaluation.evaluation_helper import get_evaluation_summary_writer
When i execute the following code: python trainer_preid.py --output=outputs/ --data=data/market/ --dataset-name=market1501 --batch-size=16 --num-epochs=100 --network-name=resnet_v1_50 --initial-checkpoint=checkpoint/resnet_v1_50.ckpt --checkpoint-exclude-scopes=resnet_v1_50/logits --trainable-scopes=resnet_v1_50/logits --no-evaluation The error occured: tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.