Exception: java.lang.VerifyError: Bad return type

[rikvdkleij]

I'm running the MNIST example and getting this error:

Exception in thread "main" java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/platanios/tensorflow/api/core/Graph.toProto()Lcom/google/protobuf/GeneratedMessageV3; @4: areturn
  Reason:
    Type 'org/tensorflow/framework/GraphDef' (current frame, stack[0]) is not assignable to 'com/google/protobuf/GeneratedMessageV3' (from method signature)
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'org/platanios/tensorflow/api/core/Graph' }
    stack: { 'org/tensorflow/framework/GraphDef' }
  Bytecode:
    0x0000000: 2ab6 04ca b0                           

    at org.platanios.tensorflow.api.core.Graph$.apply(Graph.scala:829)
    at org.platanios.tensorflow.api.core.package$.<init>(package.scala:26)
    at org.platanios.tensorflow.api.core.package$.<clinit>(package.scala)
    at org.platanios.tensorflow.api.package$.<init>(package.scala:96)
    at org.platanios.tensorflow.api.package$.<clinit>(package.scala)
    at org.platanios.tensorflow.data.image.MNISTLoader$.extractImages(MNISTLoader.scala:96)
    at org.platanios.tensorflow.data.image.MNISTLoader$.load(MNISTLoader.scala:72)

I have installed protobuffer 3.5:

protoc --version
libprotoc 3.5.1

Using 0.1.2-snapshot version and OSX.

Any idea how I can solve this?

[rikvdkleij]

Solved it by building tensorflow from sources and putting the shared libraries in LD_LIBRARY_PATH. Is this problem caused by different/incompatible protobuffer versions?

[eaplatanios]

@rikvdkleij I haven't seen this error before. Could you please send me your build.sbt file and the minimal Scala file that causes this? Thanks! :)

[rikvdkleij]

I had this issue in a large project with a lot of dependencies. A simple project with no other dependencies just works, so I suspect this problem is caused by multiple protobuf libraries on the classpath.

I will continue Monday and let you know.

Btw, it was not really solved by using TF which was build from sources. It was just working "by accident" 😄

[rikvdkleij]

Okay, the problem is caused by the BigDL library but I do not understand why... When I remove this dependency it works.

I have tried to exclude dependencies but it does not help. I expected another protobuf version on the classpath.

[eaplatanios]

@rikvdkleij Thanks for digging into this! I assume that BigDL is pulling in some other version of protobuf as you say. It may be that one of its dependencies is pulling a difference version of org.tensorflow.proto. I'm not sure. When you include the BigDL dependency, what is there version of the protobuf library that it pulls?

[rikvdkleij]

@eaplatanios BigDL does not depends on org.tensorflow.proto. I also see only version, 1.5.0.

[eaplatanios]

@rikvdkleij What about com.google.protobuf which org.tensorflow.proto depends on?

[rikvdkleij]

That is the protobuf library I already mentioned. When I make sure there is only one version in the dependency tree, the problem still occurs.

[rikvdkleij]

Found the cause, BigDL has shaded the protobuf library.

[eaplatanios]

@rikvdkleij Thanks a lot for digging into this! How do you resolve the issue in this case? Do you add a separate explicit dependency on the protobuf library?

[rikvdkleij]

No, I did not solve this problem. I only found the cause. I will leave it for later when I'm know for sure I want to use those 2 libraries in one project. I think the only solution is to fork/clone the BigDL project and rebuild it without protobuf.

Do you add a separate explicit dependency on the protobuf library?

That does not help.

[eaplatanios]

@rikvdkleij I just looked into this and I believe that we can't really do much about this in our side. jIt's weird they use the same package names for org.tensorflow.framework even though they don't keep them updated (theirs are now 9 months old). I'm wondering if there is way to specify priorities in the dependencies in SBT so that our dependency overrides their classes. I'm phrasing this a bit awkwardly but I hope this makes sense. :)

[rikvdkleij]

The only "solution" I saw was to do something like which is described here: https://stackoverflow.com/questions/7819066/sbt-exclude-class-from-jar