Closed DewaldHoman closed 5 years ago
When you say freezing, does it stay frozen? Because otherwise it's a normal behavior to pause while evaluation is being run.
Yes, it stays frozen indefinitely. But it seems to only hang after a few epochs on the first valid
batch evaluation that is passed to classify_op
during the tf session.
I'm not sure how to go about this given I cannot reproduce it. Did you find anything new?
May be this is a problem specific to your machine. Closing since I cannot reproduce.
When using mixmatch.py to train a dataset (e.g. cifar10 README example), the summary generated by
eval_stats
seems to freeze the program after a few epochs. The problem seems to be introduced at the start of evaluating thevalid
subset, during the loop:for subset in ('train_labeled', 'valid', 'test'):
images, labels = self.tmp.cache[subset]
predicted = []
for x in range(0, images.shape[0], batch):
p = self.session.run(
classify_op,
feed_dict={
self.ops.x: images[x:x + batch],
**(feed_extra or {})
})
predicted.append(p)
predicted = np.concatenate(predicted, axis=0)
accuracies.append((predicted.argmax(1) == labels).mean() * 100)
However, it does continue training at the correct epoch when interrupting and running the command again. (I am using Tensorflow 1.14 on a Titan X gpu.)