Epoch 1/2
2018-10-31 19:09:12.162738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.76GiB
2018-10-31 19:09:12.290346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:02:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-10-31 19:09:12.291317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0, 1
2018-10-31 19:09:12.750668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-31 19:09:12.750702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0 1
2018-10-31 19:09:12.750707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N Y
2018-10-31 19:09:12.750710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1: Y N
2018-10-31 19:09:12.751100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10404 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-31 19:09:12.855072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10407 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
16/16 [==============================] - 3s 201ms/step - loss: 0.0000e+00 - categorical_accuracy: 0.0000e+00
Epoch 2/2
16/16 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - categorical_accuracy: 0.0000e+00
WARNING:tensorflow:TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers).
Traceback (most recent call last):
File "Loading_error_example.py", line 47, in <module>
load_model("/home/svdvoort/test_model.hdf5")
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/engine/saving.py", line 230, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/engine/saving.py", line 310, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/layers/serialization.py", line 64, in deserialize
printable_module_name='layer')
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/utils/generic_utils.py", line 173, in deserialize_keras_object
list(custom_objects.items())))
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/engine/network.py", line 1292, in from_config
process_layer(layer_data)
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/engine/network.py", line 1278, in process_layer
layer = deserialize_layer(layer_data, custom_objects=custom_objects)
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/layers/serialization.py", line 64, in deserialize
printable_module_name='layer')
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/utils/generic_utils.py", line 175, in deserialize_keras_object
return cls.from_config(config['config'])
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/engine/base_layer.py", line 1617, in from_config
return cls(**config)
File "/packages/tensorflow/1.11.0/Python-3.6.6/tensorflow/python/keras/layers/advanced_activations.py", line 310, in __init__
if max_value is not None and max_value < 0.:
TypeError: '<' not supported between instances of 'dict' and 'float'
Same is true for all other models that are being trained
System information
Minimal working example code (Running with Python 3.6.6):
Describe the problem
Returns the following output (error included):
Same is true for all other models that are being trained
Source code / logs
Output of tf_env attached tf_env.txt