2022-03-27 16:28:09.999279: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at iterator_ops.cc:867 : Cancelled: Operation was cancelled
Traceback (most recent call last):
File "/home/hanwenxing/mobilenet_v3/train.py", line 113, in
main(args)
File "/home/hanwenxing/mobilenet_v3/train.py", line 81, in main
callbacks=callbacks,
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 727, in fit
use_multiprocessing=use_multiprocessing)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 675, in fit
steps_name='steps_per_epoch')
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 300, in model_iteration
batch_outs = f(actual_inputs)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3476, in call
run_metadata=self.run_metadata)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1472, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expected dimension in the range [0, 0), but got -1
[[{{node metrics/acc/ArgMax}}]]
(1) Invalid argument: Expected dimension in the range [0, 0), but got -1
[[{{node metrics/acc/ArgMax}}]]
[[metrics/acc/Identity/_941]]
0 successful operations.
0 derived errors ignored.
2022-03-27 16:28:09.999279: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at iterator_ops.cc:867 : Cancelled: Operation was cancelled Traceback (most recent call last): File "/home/hanwenxing/mobilenet_v3/train.py", line 113, in
main(args)
File "/home/hanwenxing/mobilenet_v3/train.py", line 81, in main
callbacks=callbacks,
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 727, in fit
use_multiprocessing=use_multiprocessing)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 675, in fit
steps_name='steps_per_epoch')
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 300, in model_iteration
batch_outs = f(actual_inputs)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3476, in call
run_metadata=self.run_metadata)
File "/home/hanwenxing/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1472, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expected dimension in the range [0, 0), but got -1
[[{{node metrics/acc/ArgMax}}]]
(1) Invalid argument: Expected dimension in the range [0, 0), but got -1
[[{{node metrics/acc/ArgMax}}]]
[[metrics/acc/Identity/_941]]
0 successful operations.
0 derived errors ignored.
Process finished with exit code 1