Deserialization exception thrown when loading a Keras SavedModel

[SchoenTannenbaum]

Description

Hi, I recently started using TensorFlow.NET and was trying to load my saved Keras model, but I get an exception thrown during model deserialization.

Specifically:

1. I saved my original pretrained .h5 Keras model to a .PB based SavedModel directory (following the instructions provided by @AsakusaRinne in this closed issue ) using a Python script running TF 2.15.0
2. I then tried to load it in C# using: var model = tf.keras.models.load_model("TestModel", false);

However, I get this error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> Newtonsoft.Json.JsonSerializationException: Could not create an instance of type Tensorflow.Keras.IRegularizer. Type is an interface or abstract class and cannot be instantiated. Path 'bias_regularizer.class_name'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject[T]()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Tensorflow.Keras.Utils.generic_utils.deserialize_keras_object(String class_name, JToken config) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Utils\generic_utils.cs:line 72
   at Tensorflow.Keras.Saving.KerasObjectLoader._revive_layer_or_model_from_config(KerasMetaData metadata, Int32 node_id) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\KerasObjectLoader.cs:line 503
   at Tensorflow.Keras.Saving.KerasObjectLoader._revive_from_config(String identifier, KerasMetaData metadata, Int32 node_id) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\KerasObjectLoader.cs:line 422
   at Tensorflow.Keras.Saving.KerasObjectLoader._load_layer(Int32 node_id, String identifier, String metadata_json) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\KerasObjectLoader.cs:line 389
   at Tensorflow.Keras.Saving.KerasObjectLoader.load_layers(Boolean compile) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\KerasObjectLoader.cs:line 79
   at Tensorflow.Keras.Saving.SavedModel.KerasLoadModelUtils.load(String path, Boolean compile, LoadOptions options) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\SavedModel\load.cs:line 62
   at Tensorflow.Keras.Saving.SavedModel.KerasLoadModelUtils.load_model(String filepath, IDictionary`2 custom_objects, Boolean compile, LoadOptions options) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Saving\SavedModel\load.cs:line 31
   at Tensorflow.Keras.Models.ModelsApi.load_model(String filepath, Boolean compile, LoadOptions options) in C:\...\TensorFlow.NET\src\TensorFlowNET.Keras\Models\ModelsApi.cs:line 13
   ...etc.

Stepping through the source (cloned from the project branch at commit 8775b0b), the exception seem to be thrown on Line 40 of ConvolutionalArgs.cs during deserialization of config for my model's bias_regularizer object (of class L1L2) from its second layer (of class Conv2D): I've tried to step further in TensorFlow.Keras and TensorFlow.Binding but I cannot find the cause.

The model I used is here: TestModel.zip

Is there something I need to specify when I save to the SavedModel in my Python script to avoid this exception when loading in C#?

Or is this something I need to do while using the Tensorflow.NET bindings to let it ingest this model without issues?

Any help for this would be greatly appreciated. Thank you!

[AsakusaRinne]

Please take a look into https://github.com/SciSharp/TensorFlow.NET/blob/8775b0b87adf5462c51a6a4340fdbf99d2e670e9/src/TensorFlowNET.Keras/Utils/generic_utils.cs#L62

and

https://github.com/SciSharp/TensorFlow.NET/blob/8775b0b87adf5462c51a6a4340fdbf99d2e670e9/src/TensorFlowNET.Keras/Utils/generic_utils.cs#L81

In this method, we use reflection to create an object according to the loaded class name and args. Generally you will see a class name related with IRegularizer here.

Besides, since your error is related with json serialization, please also look into this folder, which contains many json converters. Maybe you need to add a similar converter of IRegularizer.

I feel sorry but I don't have enough time to write the code and fix this issue recently. However, if you would like to dig into it, I'd like to help you with your questions here.

[SchoenTannenbaum]

Hi @AsakusaRinne,

Thank you for your response and for helping!

Like you mentioned, the error is coming from a call to the method deserialize_keras_object(string class_name, JToken config) , with the exception thrown specifically on this line: https://github.com/SciSharp/TensorFlow.NET/blob/8775b0b87adf5462c51a6a4340fdbf99d2e670e9/src/TensorFlowNET.Keras/Utils/generic_utils.cs#L72

I'm not to familiar with how reflection is done here, but in trying to construct Tensorflow.Keras.ArgsDefinition.Conv2DArg via Invoke() with the config as param, it chokes when going through this JSON object from inside the config:

"bias_regularizer": {
  "class_name": "L1L2",
  "config": {
    "l1": 9.999999747378752E-05,
    "l2": 0.0
  },
  "shared_object_id": 3
}

What kind of custom JsonConverter should I put together to get it to create the required C# object without problems? How do these custom converters in that folder get called?

[AsakusaRinne]

Hi, please add a converter named CustomizedRegularizerJsonConverter. You could refer to the implementation of CustomizedIinitializerJsonConverter. You need to implement two methods, which are WriteJson and ReadJson.

Don't worry, since there's only three regularizers now (L1, L2 and L1L2), when reading the json, you only need to switch the class name and create corresponding object for it. When writing the json, you have two choices. One is to switch the object type and write to json object manually, the other is to add properties with [JsonProperty(NAME)] to L1, L2 and L1L1 classes.

After adding the converter, please add an attribute [JsonConverter(typeof(CustomizedRegularizerJsonConverter))] to IRegularizer.

[SchoenTannenbaum]

Thanks!

I noticed that in Project Tensorflow.Keras there are already L1, L2, and L1L2 classes that implement IRegularizer. Should these be moved to Project Tensorflow.Bindings in somewhere like namespace Tensorflow.Operations.Regularizers and modified there instead?

As well, are the operations already implemented in the Apply() methods of the L1, L2, and L1L2 classes in Project Tensorflow.Keras correct?

[AsakusaRinne]

I noticed that in Project Tensorflow.Keras there are already L1, L2, and L1L2 classes that implement IRegularizer. Should these be moved to Project Tensorflow.Bindings in somewhere like namespace Tensorflow.Operations.Regularizers and modified there instead?

They are supposed to be in the keras package. :)

As well, are the operations already implemented in the Apply() methods of the L1, L2, and L1L2 classes in Project Tensorflow.Keras correct?

TBH I've never used them. That's why the serialization and deserialization of them were left not implemented... They were added before I began to contribute to Tensorflow.NET. I'll help you if you find the implementation incorrect.

[SchoenTannenbaum]

They are supposed to be in the keras package. :)

I guess I'm a bit confused how things are organized and the naming of the projects...

IRegularizer is in namespace Tensorflow.Keras of Project Tensorflow.Bindings, though the L1, L2, and L1L2 classes of Project Tensorflow.Keras through its dependency on Project Tensorflow.Bindings refer that interface .

I may very well be implementing something wrong... but since Project Tensorflow.Bindings does not depend on Project Tensorflow.Keras, rather than either duplicating the classes on both projects, Project Tensorflow.Keras could just refer to the classes if we move them there?

TBH I've never used them. That's why the serialization and deserialization of them were left not implemented... They were added before I began to contribute to Tensorflow.NET. I'll help you if you find the implementation incorrect.

I'll see if the regularizer operations are correct. Hopefully the implementer checked how it was done in the original.

[AsakusaRinne]

I may very well be implementing something wrong... but since Project Tensorflow.Bindings does not depend on Project Tensorflow.Keras, rather than either duplicating the classes on both projects, Project Tensorflow.Keras could just refer to the classes if we move them there?

Since Tensorflow.Keras depends on Tensorflow.Binding, a recycling reference will be detected if making Tensorflow.Keras of one of Tensorflow.Binding's dependency. The reason of putting IRegularizer inTensorflow.Bindingis because it's needed inTensorflow.Binding`.

TBH I don't like this design, which is from the creator of tf.net. I prefer to include Tensorflow.Keras in Tensorflow.Binding so that it won't be confusing. However, it will make huge break change so at this time we still need to follow the design...

[SchoenTannenbaum]

a recycling reference will be detected if making Tensorflow.Keras of one of Tensorflow.Binding's dependency.

I understand that Project Tensorflow.Keras should not be one of Project Tensorflow.Binding's dependencies. What I'm saying is that since the classes in Project Tensorflow.Keras are already using IRegularizer there and the dependency is already there, they might as well be move so that we don't duplicate these classes.

[AsakusaRinne]

I understand what you are talking about. However, L1L2 regularizer seems to be a part of keras instead of tensorflow itself. Tensorflow.NET is a binding for tensorflow so I think we should either do nothing or merge these two packages (making keras a namespace of tf.net package).

[SchoenTannenbaum]

So where should we locate the regularizer classes? Tensorflow.Binding needs L1L2 in its project to deserialize these objects, which would mean that if we don't move them from the Tensorflow.Keras projects we would have to 1) do a circular reference or 2) make the same copy in both projects. Both these ways are not ideal.

I'm looking at the TF 2.15 doc and though L1L2 is in Keras, it is also part of the TF API https://www.tensorflow.org/versions/r2.15/api_docs/python/tf/keras/regularizers/L1L2

Incidentally, the operations in the Apply() methods in one of the regularizers and one of the default values another are wrong. They're a straightforward fixs and I've done them on my side.

[AsakusaRinne]

Oh, sorry, you're right. I forgot that the class is required in the json converter. Then please put it in Tensorflow.Operations is fine. :)

[SchoenTannenbaum]

Thanks! Seems to work now!

[AsakusaRinne]

Would you like to make a PR for your fix? That will help others who have the same problem in the future.

[SchoenTannenbaum]

For sure! I put something together tomorrow.

[SchoenTannenbaum]

BTW are coded unit tests required for PR to this project? If not, then : https://github.com/SciSharp/TensorFlow.NET/pull/1248

[AsakusaRinne]

Yes, but a simple one is enough. You could either add your model files to this asset and add a simple unit test like this one https://github.com/SciSharp/TensorFlow.NET/blob/master/test/TensorFlowNET.Keras.UnitTest/Model/ModelLoadTest.cs#L68

[SchoenTannenbaum]

I modded ModelSaveTest to write out the bias_regularizer which ModelLoadTest read in. It passes.

[SchoenTannenbaum]

Messed something on my machine and had to redo everything. Please see PR at https://github.com/SciSharp/TensorFlow.NET/pull/1250