TF: No way to create a string tensor with arbitrary binary data?

[madprogrammer]

Description

I'm trying to feed a Google AutoML Object detection saved_model.pb, which accepts a JPEG/PNG-encoded file as DT_STRING(-1) tensor on the input. It worked well in Java TensorFlow, but I'm struggling to make it work in DJL. Can't find a way to create binary tensor with type String. After few hours of reading DJL code, tried the following, without success:

    @Override
    public NDList processInput(TranslatorContext ctx, MatOfByte input) {
        NDArray imageBytes = ctx.getNDManager().create(ByteBuffer.wrap(input.toArray()), new Shape(), DataType.STRING);
        imageBytes.setName("image_bytes");

        NDArray key = ctx.getNDManager().create("test");
        key.setName("key");

        return new NDList(imageBytes, key);
    }

The same saved_model.pb works fine from Java TensorFlow API, but with DJL TensorFlow complains, and the tensor handle in imageBytes seems to have tensor with address = 0 which seems not right to me (see screenshot).

W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at strided_slice_op.cc:108 : Invalid argument: slice index 0 of dimension 0 out of bounds.
org.tensorflow.exceptions.TFInvalidArgumentException: slice index 0 of dimension 0 out of bounds.

The shape of the AutoML model:

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['image_bytes'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: encoded_image_string_tensor:0
    inputs['key'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: key:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 40, 4)
        name: detection_boxes:0
    outputs['detection_classes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 40)
        name: detection_classes:0
    outputs['detection_classes_as
<img width="651" alt="idea64_a63JYaAaR7" src="https://user-images.githubusercontent.com/531058/134743318-cfd700ab-4aed-4721-8346-661f2104df5e.png">
_text'] tensor_info:
        dtype: DT_STRING
        shape: (-1, -1)
        name: detection_classes_as_text:0
    outputs['detection_multiclass_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 40, 3)
        name: detection_multiclass_scores:0
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 40)
        name: detection_scores:0
    outputs['image_info'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 6)
        name: Tile_1:0
    outputs['key'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Identity:0
    outputs['num_detections'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: num_detections:0
  Method name is: tensorflow/serving/predict

Working code from Java TensorFlow API to create the input tensors:

TString inputTensor = TString.tensorOfBytes(NdArrays.vectorOfObjects(buffer.toArray()));
TString keyTensor = TString.scalarOf("test");

Expected Behavior

There is a way to create a binary string tensor, as it's possible with TensorFlow Java, Python and other APIs

Error Message

 org.tensorflow.exceptions.TFInvalidArgumentException: slice index 0 of dimension 0 out of bounds.
     [[{{node map/TensorArrayUnstack/strided_slice}}]]
    at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:87)
    at ai.djl.tensorflow.engine.javacpp.JavacppUtils.runSession(JavacppUtils.java:192)
    at ai.djl.tensorflow.engine.TfSymbolBlock.forwardInternal(TfSymbolBlock.java:131)
    at ai.djl.nn.AbstractBlock.forward(AbstractBlock.java:121)
    at ai.djl.nn.Block.forward(Block.java:122)
    at ai.djl.inference.Predictor.predict(Predictor.java:123)
    at ai.djl.inference.Predictor.batchPredict(Predictor.java:150)
    at ai.djl.inference.Predictor.predict(Predictor.java:118)

What have you tried to solve it?

Spent hours of reading the code of TfNDManager, JavacppUtils etc. to try to understand how to do it.

Environment Info

DJL version: 0.12.0

[frankfliu]

@madprogrammer

Do you mind share your project?

[madprogrammer]

@frankfliu I've isolated the case to a small project forked from the djl-demo/quarkus/example https://github.com/madprogrammer/djl-issue-prj

It can be run by executing mvnw quarkus:dev, and then going to http://127.0.0.1:8080/detect

[madprogrammer]

I found that the created tensor is deallocated as soon as the PointerScope block is exited

I'm not sure but somehow this doesn't seem right to me.

[madprogrammer]

After some more digging and comparison to TensorFlow Java API, I found that the working code from TensorFlow Java creates a DT_STRING tensor with shape [1], and numBytes = Loader.sizeof(TF_TString.class), and apparently there is no way to do it with the current API present in TfNDManager and JavacppUtils

[frankfliu]

@madprogrammer Currently TfNDManager.create(String[]) was not implemented in DJL, you cannot create String tensor other than scalar. But you should be easily get shape (1) tensor by nd.expandDims(0):

        String data = new String(input.array(), StandardCharsets.UTF_8);
        NDArray imageBytes = ctx.getNDManager().create(data).expandDims(0);
        imageBytes.setName("image_bytes");

[frankfliu]

@madprogrammer I created PR that allows you to create String tensor with specified shape: #1251

[madprogrammer]

@frankfliu Thank you, but the problem is also that calling String data = new String(input.array(), StandardCharsets.UTF_8) will corrupt the binary data after decoding back to byte array in JavacppUtils, or even probably throw an exception in case an invalid byte sequence for UTF-8 encoding is encountered. TensorFlow defines DT_STRING as variable-length byte arrays. There probably should be an overload that allows to pass a byte[] array directly down to createStringTensor in JavacppUtils, or something like that, to avoid calling str.getBytes inside JavacppUtils to make it work.

[madprogrammer]

I just tested that encoding and decoding with StandardCharsets.US_ASCII provides same byte array that was before encoding to String, but inside of JavacppUtils UTF-8 is used anyway, so it's not going to work, and looks more like a dirty hack than a good code.

[madprogrammer]

@frankfliu Implementation proposal

    @SuppressWarnings({"unchecked", "try"})
    public static Pair<TF_Tensor, TFE_TensorHandle> createStringTensor(long[] dims, String[] src) {
        ByteBuffer[] byteBuffers = new ByteBuffer[src.length];
        for (int i = 0; i < src.length; i++)
            byteBuffers[i] = ByteBuffer.wrap(src[i].getBytes(StandardCharsets.UTF_8));
        return createStringTensor(dims, byteBuffers);
    }

    @SuppressWarnings({"unchecked", "try"})
    public static Pair<TF_Tensor, TFE_TensorHandle> createStringTensor(long[] dims, ByteBuffer[] src) {
        int dType = TfDataType.toTf(DataType.STRING);
        long numBytes = (long) Loader.sizeof(TF_TString.class) * src.length;
        try (PointerScope ignored = new PointerScope()) {
            /*
             * String tensor allocates a separate TF_TString memory. The TF_TString will
             * be deleted when the string tensor is closed. We have to track TF_TString
             * memory by ourselves and make sure thw TF_TString lifecycle align with
             * TFE_TensorHandle. TF_Tensor already handles TF_TString automatically, We
             * can just keep a TF_Tensor reference in TfNDArray.
             */
            TF_Tensor tensor = AbstractTF_Tensor.allocateTensor(dType, dims, numBytes);
            Pointer pointer = tensorflow.TF_TensorData(tensor).capacity(numBytes);
            TF_TString data = new TF_TString(pointer).capacity(pointer.position() + src.length);
            for (int i = 0; i < src.length; ++i) {
                TF_TString tstring = data.getPointer(i);
                tensorflow.TF_TString_Copy(tstring, new BytePointer(src[i]), src[i].remaining());
            }

            TF_Status status = TF_Status.newStatus();
            TFE_TensorHandle handle = AbstractTFE_TensorHandle.newTensor(tensor, status);
            status.throwExceptionIfNotOK();

            handle.retainReference();
            tensor.retainReference();
            return new Pair<>(tensor, handle);
        }
    }

[frankfliu]

@madprogrammer Thanks for your inputs.

Now you can create String tensor using either charset or use ByteBuffer:

        TfNDManager manager = ((TfNDManager)ctx.getNDManager());
        NDArray imageBytes = manager.createStringTensor(new Shape(1), ByteBuffer.wrap(input));
        imageBytes.setName("image_bytes");

The ByteBuffer only available in TfNDManager.

[frankfliu]

This is fixed by https://github.com/deepjavalibrary/djl/pull/1251. You can try 0.13.0-SNAPSHOT version now.