Open robert-hardwick opened 4 years ago
Have debugged further
input_1
Conv_pad
Conv
Conv/BatchNorm
activation
expanded_conv/depthwise/pad
expanded_conv/depthwise
Traceback (most recent call last):
File "concatenate_networks.py", line 97, in <module>
main()
File "concatenate_networks.py", line 58, in main
mobilenet_model.save(cmd_args.out_saved_model)
File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/engine/network.py", line 1007, in save
save.save_model(self, filepath, overwrite, include_optimizer, save_format,
File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/save.py", line 111, in save_model
hdf5_format.save_model_to_hdf5(
File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
save_weights_to_hdf5_group(model_weights_group, model_layers)
File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 626, in save_weights_to_hdf5_group
g = f.create_group(layer.name)
File "/home/robert/tf2/lib/python3.8/site-packages/h5py/_hl/group.py", line 68, in create_group
gid = h5g.create(self.id, name, lcpl=lcpl, gcpl=gcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5g.pyx", line 161, in h5py.h5g.create
ValueError: Unable to create group (name already exists)
Appears that it doesn't like the layer name "depthwise/pad" before the parent group "depthwise" https://github.com/keras-team/keras-applications/blob/bc89834ed36935ab4a4994446e34ff81c0d8e1b7/keras_applications/mobilenet_v3.py#L161-L162
Guess that's something to do the h5 heirarchical data structure.
Renaming depthwise layer here https://github.com/keras-team/keras-applications/blob/bc89834ed36935ab4a4994446e34ff81c0d8e1b7/keras_applications/mobilenet_v3.py#L163-L167 to "depthwise/DepthwiseConv" fixes the error mentioned above, however the predictions are still different compared to saving as SavedModel.
Might be 2 separate issues here.
Again, the above debug was using tensorflow keras, had assumed that keras was silently failing but since fixing the layer naming the problem still persists with same symptoms as kera. Will use keras for further debugging.
Summary
Created an instance of MobileNetV3Small, attempted to save model with weights to .h5 file and after loading back again the model gives a different output or with tensorflow.keras fails with error.
Issue not present with save_format="tf" / SavedModel
In tensorflow.keras this gave error of: ValueError: Unable to create group (Name already exists)
After reading about this error it leads me to believe there is an issue with h5 file format and layers with the same name. I tried the same code but with mobilenet_v2 and there is no problem, so my conclusion is that the mobilenetv3 implementation is using duplicated layer names somewhere.
Environment
Logs or source codes for reproduction
Code :
When using tensorflow.keras
When using keras ( It fails silently and simply outputs incorrect values )
Notice 0.98741907 != 0.917123