JNI Error: Tensor Creation from ByteBuffer

[mandar2812]

So I'm using the Tensor.fromBuffer method for loading data from a number of images and I get the following JNI error. (Condensed form of the error dump)

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe44f565251, pid=10465, tid=0x00007fe4ff796700
#
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libtensorflow.so+0x4c2251]  TF_TensorData+0x1
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

The source of the error is

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.platanios.tensorflow.jni.Tensor$.fromBuffer(I[JJLjava/nio/ByteBuffer;)J+0
j  org.platanios.tensorflow.api.tensors.Tensor$.fromBuffer(Lorg/platanios/tensorflow/api/types/DataType;Lorg/platanios/tensorflow/api/core/Shape;JLjava/nio/ByteBuffer;)Lorg/platanios/tensorflow/api/tensors/Tensor;+96
j  io.github.mandar2812.dynaml.tensorflow.package$dtf$.tensor_from_buffer(Ljava/lang/String;Lscala/collection/Seq;[B)Lorg/platanios/tensorflow/api/tensors/Tensor;+27

Source

In the relevant source code I see the following TODO note

@throws[IllegalArgumentException]
  def fromBuffer(dataType: DataType, shape: Shape, numBytes: Long, buffer: ByteBuffer): Tensor = this synchronized {
    // TODO: May behave weirdly for direct byte buffers allocated on the Scala side.
    val directBuffer = {
      if (buffer.isDirect) {
        buffer
      } else {
        val direct = ByteBuffer.allocateDirect(numBytes.toInt)
        val bufferCopy = buffer.duplicate()
        direct.put(bufferCopy.limit(numBytes.toInt).asInstanceOf[ByteBuffer])
        direct
      }
    }
    val hostHandle = NativeTensor.fromBuffer(dataType.cValue, shape.asArray.map(_.toLong), numBytes, directBuffer)
    val tensor = Tensor.fromHostNativeHandle(hostHandle)
    NativeTensor.delete(hostHandle)
    tensor
  }

in DynML I use the following code

def tensor_from_buffer[T](
      dtype: DataType.Aux[T], shape: Shape)(
      buffer: Array[Byte]): Tensor = {
      Tensor.fromBuffer(dtype, shape, buffer.length.toLong, ByteBuffer.wrap(buffer))

So I think here the byte buffer is being allocated on the Scala side, no?

[eaplatanios]

Could you try again using the latest 0.3.0-SNAPSHOT and the precompiled binaries distributed with TF Scala? It currently works using the master branch of TensorFlow (i.e., the nightly builds). I finally got what we needed from the TF team. Sorry for the big delay in responding but I've been waiting for that for a while now. :)

[eaplatanios]

@mandar2812 I don't seem able to reproduce this in the current master (0.4.0-SNAPSHOT) and so I'll close this for now. Please let me know if the issue persists.