Variable Initialization Overrides All Other Variables

[violetteavi]

Hello,

I've been working on reconstructing the MNIST neural net found here using the TensorFlowSharp 1.3.0 nuget package in VisualStudio 2015

I was receiving several Matrix size-incompatible exceptions when I tried to add two matrices of the same dimensions. After some tinkering, I discovered that one of my variables was being overwritten on initialization with the shape of another variable. Here is some example code highlighting the issue:

using (var graph = new TFGraph())
{
   TFOutput variableOneShape = graph.Const(new TFShape(784, 30).ToIntArray());
   TFOutput initialVariableOne = graph.RandomUniform(variableOneShape, TFDataType.Float);
   TFOutput variableOne;
   TFOperation variableOneInit;
   TFOutput persistentVariableOne = graph.Variable(initialVariableOne, out variableOneInit, out variableOne, "Var1");
   TFOutput variableTwoShape = graph.Const(new TFShape(1, 30).ToIntArray());
   TFOutput initialVariableTwo = graph.RandomUniform(variableTwoShape, TFDataType.Float);
   TFOutput variableTwo;
   TFOperation variableTwoInit;
   TFOutput persistentVariableTwo = graph.Variable(initialVariableTwo, out variableTwoInit, out variableTwo, "Var2");

   using (TFSession session = new TFSession(graph))
   {
       session.GetRunner().AddTarget(variableOneInit).Run();
       session.GetRunner().AddTarget(variableTwoInit).Run();
       TFSession.Runner debugFetcher = session.GetRunner()
                                     .Fetch(initialVariableOne)
                                     .Fetch(variableOne)
                                     .Fetch(initialVariableTwo)
                                     .Fetch(variableTwo);
       TFTensor[] result = debugFetcher.Run();
       TFTensor returnedInitialVariableOne = result[0];
       TFTensor returnedVariableOne = result[1];
       TFTensor returnedInitialVariableTwo = result[2];
       TFTensor returnedVariableTwo = result[3];
  }
}

Depending on the order of initialization operation, either returnedVariableOne or returnedVariableTwo are overwritten with the shape of the other. A similar thing occurs with graph.GetGlobalVariablesInitializer()

[violetteavi]

This also appears to happen when merely setting a variable with AssignVariableOp. Am I missing something?

[cesarsouza]

Hi @violetteavi,

I suppose I am running into the same issue. Have you found any solution or hints on what may be causing the issue? If not, I may try to dig a bit into this part of the code to see what's happening.

Regards, Cesar

[cesarsouza]

Hi @violetteavi,

I've managed to sidestep this issue by using the VariableV2 method instead of Variable. I am adding below two unit tests that might be useful in the project if anyone would like to continue investigating this issue. The first uses the Variable() method and fails. the second uses VariableV2 and succeeds.

[Fact]
public void MultipleVariables_InitializedSeparately_HaveSeparateValues ()
{
    using (var graph = new TFGraph ()) {
        TFOperation initW;
        TFOutput valueW;
        var W = graph.Variable (graph.Const (1.0), out initW, out valueW, operName: "W");

        TFOperation initb;
        TFOutput valueb;
        var b = graph.Variable (graph.Const (-0.3), out initb, out valueb, operName: "b");

        var y = graph.Add(valueW, valueb);

        using (var sess = new TFSession (graph)) {

            Assert.Throws<TFException> (() => sess.GetRunner ().Fetch (y).Run ()); // ok

            var r1 = sess.GetRunner ().AddTarget (initW).Run ();
            var r2 = sess.GetRunner ().AddTarget (initb).Run ();
            var r = sess.GetRunner ().Fetch (y).Run ();

            Assert.Equal (0.7, (double)r[0].GetValue ()); // fail, will return -0.6 because both vars will be initialized with 0.3
        }
    }
}

[Fact]
public void MultipleVariables_InitializedSeparately_HaveSeparateValues_V2 ()
{
    using (var graph = new TFGraph ()) {

        var W = graph.VariableV2 (TFShape.Scalar, TFDataType.Double, operName: "W");
        var initW = graph.Assign (W, graph.Const (1.0));

        var b = graph.VariableV2 (TFShape.Scalar, TFDataType.Double, operName: "b");
        var initb = graph.Assign (b, graph.Const (-0.3));

        var y = graph.Add (W, b);

        using (var sess = new TFSession (graph)) {

            Assert.Throws<TFException> (() => sess.GetRunner ().Fetch (y).Run ()); // ok

            var r1 = sess.GetRunner ().AddTarget (initW.Operation).Run ();
            var r2 = sess.GetRunner ().AddTarget (initb.Operation).Run ();
            var r = sess.GetRunner ().Fetch (y).Run ();

            Assert.Equal (0.7, (double)r [0].GetValue ()); // ok
        }
    }
}

Regards, Cesar

[violetteavi]

Ok, thanks for confirming that I'm not crazy lol. What version has VariableV2? TFS 1.3 only has Variable.

Also I have not looked at the source code yet, will let you know when I do.

[cesarsouza]

I am using the latest pre-release version of TensorFlowSharp on NuGet (1.4.0-pre1).

Now that I've sidestepped this I've finally managed to get MLPs to train using SGD and Adam. If you are interested in eventually using those, I am going to update Keras Sharp with those implementations and hopefully I should be able to add some examples on how to use them quite soon.