image to byte array - segment too large

[JohannesBeranek]

This might be more of a question, depending on if this is actually solvable with the current vips-ffm api.

JVips:

    @Override
    public byte[] writeJpegImage(VipsImage image, float quality) {
        try {
            image.applyIccTransform("srgb", new ICCTransformOptions().embedded(true));
        } catch(VipsException ignored) {}

        return image.writeJPEGToArray(toVipsQuality(quality), true);
    }

This works for all our images, independent of size.

vips-ffm:

    @Override
    public byte[] writeJpegImage(VImage image, float quality) {
        VBlob blob = image
            .iccTransform("srgb", VipsOption.Boolean("embedded", true))
            .jpegsaveBuffer(VipsOption.Int("Q", toVipsQuality(quality)));

        return blob.getUnsafeAddress().toArray(ValueLayout.JAVA_BYTE);
    }

This works up to a certain image size.

For larger images though, it produces the following:

Caused by: java.lang.IllegalStateException: Segment is too large to wrap as byte[]. Size: 9223372036854775807
        at jdk.internal.foreign.AbstractMemorySegmentImpl.checkArraySize(AbstractMemorySegmentImpl.java:396) ~[?:?]
        at jdk.internal.foreign.AbstractMemorySegmentImpl.toArray(AbstractMemorySegmentImpl.java:352) ~[?:?]
        at jdk.internal.foreign.AbstractMemorySegmentImpl.toArray(AbstractMemorySegmentImpl.java:318) ~[?:?]
        at some.package.SomeClass.writeJpegImage(SomeClass.java:80) ~[app_B:?]

What would be the correct way to get a byte array from an image, without writing it out to disk and reading it in again?

[JohannesBeranek]

Looking at the reported size, it looks to me like either a NUL termination byte is missing at the end of the blob, or I need to get the length somewhere to limit the used blob size, because a jpeg image is definitely not around 9 petabyte.

[lopcode]

I think it's reasonable to want a safe (no raw pointers) and efficient (no copying) way to get at raw VBlob (VipsArea) data, so I've added a couple of new methods to VBlob in 0.5.8:

• VBlob.asByteBuffer() - recommended way to get at the bytes, should be directly mapped to native memory for efficiency, and you can still access its byte[] array from there if you need it (though the raw array might require an explicit copy, I think not all ByteBuffers necessarily implement array() - this isn't any worse than JVips though which I believe always makes a copy)
• VBlob.byteSize() - if you're sure you just want the byte length of the data, grabbed right out of the VipsArea struct
• VBlob.getUnsafeDataAddress() - if you really want access to the raw bytes - the MemorySegment is sliced to the length of the data, so it's safer even if the data isn't null-terminated

[JohannesBeranek]

@lopcode this seems to work now (0.5.9), thanks! I think it would be good to add to docs/readme/javadoc/... that the ByteBuffer returned by these is a DirectByteBuffer pointing to the memory inside the Arena, and thus should not be accessed once the arena goes out of scope (I just had to debug an issue with that, and have now switched back to returning only byte[] from our final write* methods to make sure there'll be no issue with that).

[lopcode]

Yep agree about the docs - I made some improvements:

• Renamed VipsBlob.asByteBuffer to VipsBlob.asArenaScopedByteBuffer
• Added VipsBlob.asClonedByteBuffer
• Made docs clearer for both