[BUG Report]: Unable to load multiple models at once

[clarson15]

Description

I am unable to create multiple networks at the same time. I have a file containing the weights of my network, but only 1 model.load_weights succeeds. I thought maybe it was unable to load the same file multiple times, so I tried loading different files and still only the first model succeeds. I am getting the error message:

Exception thrown: 'Tensorflow.ValueError' in Tensorflow.Keras.dll: 'You are trying to load a weight file containing System.String[] layers into a model with 5 layers.'
   at Tensorflow.Keras.Saving.hdf5_format.load_weights_from_hdf5_group(Int64 f, List`1 layers)
   at Tensorflow.Keras.Engine.Model.load_weights(String filepath, Boolean by_name, Boolean skip_mismatch, Object options)

Reproduction Steps

public class Network {
    private IModel model;
    public Network()
    {
        var layers = keras.layers;
        var input = keras.Input(shape: 301, name: "game-parameters");
        var dense = layers.Dense(256, activation: keras.activations.Relu).Apply(input);

        var output = layers.Dense(1, activation: keras.activations.Linear).Apply(dense);

        model = keras.Model(input, output, name: "example");
        model.compile(optimizer: keras.optimizers.Adam(learning_rate: 0.001f), loss: keras.losses.MeanSquaredError(), metrics: new[] { keras.metrics.Accuracy() });
    }

    public void Save()
    {
        model.save_weights("./model.h5, save_format: "h5");
    }

    public void Load()
    {
        model.load_weights("./model.h5");
    }
}

public class Program
{
    public static void Main()
    {
        var model1 = new Network();
        var model2 = new Network();
        model1.Save();
        model2.Load();

        // or if model.h5 already exists
       model1.Load();
       model2.Load();
    }
}

Known Workarounds

No response

Configuration and Other Information

No response

[clarson15]

Another issue.. Concatenate layer does not save/load correctly using save and load_model.

var layers = keras.layers;
var input1 = keras.Input(shape: (7, 7, 25), name: "map");
var conv = layers.Conv2D(30, 3, activation: keras.activations.Relu).Apply(input1);
var flatten = layers.Flatten().Apply(conv);

var input2 = keras.Input(shape: 301, name: "game-parameters");
var dense = layers.Dense(256, activation: keras.activations.Relu).Apply(input2);

var concat = layers.Concatenate().Apply((flatten, dense));
var dense2 = layers.Dense(256, activation: keras.activations.Relu).Apply(concat);
var dense3 = layers.Dense(128, activation: keras.activations.Relu).Apply(dense2);

var output = layers.Dense(1, activation: keras.activations.Linear).Apply(dense3);

model = keras.Model(inputs: (input1, input2), outputs: output, name: "example");
model.compile(optimizer: keras.optimizers.Adam(learning_rate: 0.001f), loss: keras.losses.MeanSquaredError(), metrics: new[] { keras.metrics.Accuracy() });

saving this model and then attempting to load it returns this error:

Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll: 'An item with the same key has already been added. Key: Tensorflow.Keras.Layers.Concatenate'
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException[T](T key)
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Tensorflow.Keras.Engine.Functional.process_layer(Dictionary`2 created_layers, LayerConfig layer_data, Dictionary`2 unprocessed_nodes, Dictionary`2 node_count_by_layer)
   at Tensorflow.Keras.Engine.Functional.reconstruct_from_config(FunctionalConfig config, Dictionary`2 created_layers)
   at Tensorflow.Keras.Saving.KerasObjectLoader._reconstruct_model(Int32 model_id, Model model, List`1 layers)
   at Tensorflow.Keras.Saving.KerasObjectLoader._reconstruct_all_models()
   at Tensorflow.Keras.Saving.KerasObjectLoader.finalize_objects()
   at Tensorflow.Keras.Saving.SavedModel.KerasLoadModelUtils.load(String path, Boolean compile, LoadOptions options)
   at Tensorflow.Keras.Saving.SavedModel.KerasLoadModelUtils.load_model(String filepath, IDictionary`2 custom_objects, Boolean compile, LoadOptions options)
   at Tensorflow.Keras.Models.ModelsApi.load_model(String filepath, Boolean compile, LoadOptions options)```

[clarson15]

Any update/fix for this? I'm completely blocked until this can be resolved. I'm trying to implement reinforcement learning for a game w/ benchmarking on previous versions of itself which worked fine in tensorflow.js but saving/loading seems completely broken in Tensorflow.net.

[Wanglongzhi2001]

Any update/fix for this? I'm completely blocked until this can be resolved. I'm trying to implement reinforcement learning for a game w/ benchmarking on previous versions of itself which worked fine in tensorflow.js but saving/loading seems completely broken in Tensorflow.net.

Very sorry to reply to your message so late. The implementation of model.save_weights and model.load_weights in TensorFlow.NET is not comprehensive. We recommand you to use model.save and keras.load_model to save and load the whole model, which's implementation is more perfect. Just like the following:

public void Save()
{
    model.save("./saved_model");
}

public void Load()
{
    keras.models.load_model("./saved_model");
}

When modified the code you provide, it runs successfully. If there still exist some problems, please tell me.

[clarson15]

My main problem was saving/loading was not working because I had a concat layer, which PR #1192 fixed. Without that, I was attempting to use save/load _weights as a workaround