String tensor throws "Malformed TF_STRING tensor; element 0 out of range"

[MLnick]

Came across this when trying to use tf.Saver(). Simple to reproduce using

object Repro extends App {
  tf.Saver(allowEmpty = true)
}

Exception in thread "main" java.lang.IllegalArgumentException: Malformed TF_STRING tensor; element 0 out of range
    at org.platanios.tensorflow.jni.Op$.setAttrTensor(Native Method)
    at org.platanios.tensorflow.api.ops.Op$Builder$$anonfun$org$platanios$tensorflow$api$ops$Op$Builder$$setAttributes$1.apply(Op.scala:1602)
    at org.platanios.tensorflow.api.ops.Op$Builder$$anonfun$org$platanios$tensorflow$api$ops$Op$Builder$$setAttributes$1.apply(Op.scala:1579)
    at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
    at org.platanios.tensorflow.api.ops.Op$Builder.org$platanios$tensorflow$api$ops$Op$Builder$$setAttributes(Op.scala:1579)
    at org.platanios.tensorflow.api.ops.Op$Builder$$anonfun$build$1.apply(Op.scala:1566)
    at org.platanios.tensorflow.api.ops.Op$Builder$$anonfun$build$1.apply(Op.scala:1543)
    at org.platanios.tensorflow.api.package$.using(package.scala:58)
    at org.platanios.tensorflow.api.ops.Op$Builder.build(Op.scala:1543)
    at org.platanios.tensorflow.api.ops.Basic$$anonfun$constant$1.apply(Basic.scala:83)
    at org.platanios.tensorflow.api.ops.Basic$$anonfun$constant$1.apply(Basic.scala:79)
    at org.platanios.tensorflow.api.package$.using(package.scala:58)
    at org.platanios.tensorflow.api.ops.Basic$class.constant(Basic.scala:79)
    at org.platanios.tensorflow.api.ops.Basic$.constant(Basic.scala:2998)
    at org.platanios.tensorflow.api.tensors.Tensor$class.toOpOutput(Tensor.scala:166)
    at org.platanios.tensorflow.api.tensors.StringTensor.toOpOutput(StringTensor.scala:28)
    at org.platanios.tensorflow.api.ops.variables.SaverDefBuilder$$anonfun$20.apply(Saver.scala:1029)
    at org.platanios.tensorflow.api.ops.variables.SaverDefBuilder$$anonfun$20.apply(Saver.scala:1027)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at org.platanios.tensorflow.api.ops.Op$.createWithNameScope(Op.scala:701)
    at org.platanios.tensorflow.api.ops.variables.SaverDefBuilder$class.build(Saver.scala:1027)
    at org.platanios.tensorflow.api.ops.variables.DefaultSaverDefBuilder$.build(Saver.scala:1392)
    at org.platanios.tensorflow.api.ops.variables.Saver$.apply(Saver.scala:360)
    at Repro$.delayedEndpoint$Repro$1(Test.scala:13)
    at Repro$delayedInit$body.apply(Test.scala:12)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at Repro$.main(Test.scala:12)
    at Repro.main(Test.scala)

[eaplatanios]

Thanks for finding this out! I just got back from traveling. I'll look into this today and let you know what I find. :)

[eaplatanios]

@MLnick It turns out there were a couple problems. One related to string tensors and one related to empty savers. If you use savers it would be great if you couple add a few simple unit tests based on your usage if you get the chance. In either case, thanks a lot for reporting this. :)

[MLnick]

@eaplatanios another issue seems to have cropped up:

Exception in thread "main" java.lang.IllegalArgumentException: Input tensor_names and shape_and_slices should be an 1-D tensors, got [1,1] and [1,1] instead.
     [[Node: Saver/Save = SaveV2[_class=[], dtypes=[DT_DOUBLE], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Saver/Constant_0_0, Saver/Constant_1, Saver/Constant_2, weights/weights/ReadVariable)]]
    at org.platanios.tensorflow.jni.Session$.run(Native Method)
    at org.platanios.tensorflow.api.core.client.Session.runHelper(Session.scala:127)
    at org.platanios.tensorflow.api.core.client.Session.run(Session.scala:66)
    at org.platanios.tensorflow.api.ops.variables.Saver.save(Saver.scala:176)

[MLnick]

I haven't used to saver much and am not that familiar with even the Python / native version of it. I am trying to play around with it to see if I can export graphs from this lib and re-use them in say Python or TF serving, etc. And ran into the issue.

Will see if I can put together some test cases.

[eaplatanios]

@MLnick This had to do with not managing string tensors very well, but I think it's been fixed in the last two commits.

[MLnick]

Sorry to be the bearer of bad news, but still getting an error. Seems the shape of the relevant string tensors is still not correct (I have 2 variables that are getting saved hence the [1,2] shapes, that should I think be 1-D tensors of length 2?)

Exception in thread "main" java.lang.IllegalArgumentException: Input tensor_names and shape_and_slices should be an 1-D tensors, got [1,2] and [1,2] instead.
     [[Node: Saver/Save = SaveV2[_class=[], dtypes=[DT_DOUBLE, DT_DOUBLE], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Saver/Constant_0_0, Saver/Constant_1, Saver/Constant_2, weights/weights/ReadVariable, intercept/intercept/ReadVariable)]]
    at org.platanios.tensorflow.jni.Session$.run(Native Method)
    at org.platanios.tensorflow.api.core.client.Session.runHelper(Session.scala:127)
    at org.platanios.tensorflow.api.core.client.Session.run(Session.scala:66)
    at org.platanios.tensorflow.api.ops.variables.Saver.save(Saver.scala:176)
    at TensorFlowTest$.delayedEndpoint$TensorFlowTest$1(Test.scala:53)
    at TensorFlowTest$delayedInit$body.apply(Test.scala:13)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at TensorFlowTest$.main(Test.scala:13)
    at TensorFlowTest.main(Test.scala)

[eaplatanios]

@MLnick Thanks a lot for pointing out all these issues. They only serve to point out the usefulness of unit tests. I'll try to add unit tests for ops after I get support for some more ops.

I fixed this issue, but I'll leave it open until you confirm all seems to work fine for you. :) Feel free to close it when you feel the bug is fixed.

[MLnick]

Thanks, that seems to have allowed me to save successfully. Now going to try loading a saved model and predicting with it :)

[eaplatanios]

That's great! The restoring ops are sort of symmetric to the save ops and I changed them too in my last fix, so I hope this will work fine for you. :)