get vsimem file byte array while using java binding

[sendreams]

Feature description

hi: i have saved a dataset to png file.

file = "/vsimem/output.png"; out_drv.CreateCopy(file, dstile, 0, new String[0]);

then i want to get the file's byte array and save the bytes to database.

In C++ gdal，VSIGetMemFileBuffer can convert /vsimem/** file to byte[], but I can not find corresponding method in java gdal. What can I do to use it in java?

Additional context

using gdal 3.6.3

[sendreams]

The following code works in RGB mode but does not function in ARGB mode.

private static byte[] exportToPNG(Dataset dataset) throws IOException {
        // 将内存数据集转换为BufferedImage
        int width = dataset.getRasterXSize();
        int height = dataset.getRasterYSize();
        int bands = dataset.getRasterCount();
        // 透明色无法输出 no alpha band 
//        BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
//        for (int i = 1; i <= bands - 1; i++) {
//            Band band = dataset.GetRasterBand(i);
//            int[] buffer = new int[width * height];
//            band.ReadRaster(0, 0, width, height, buffer);
//            image.getRaster().setSamples(0, 0, width, height, i-1, buffer);
//        }
        // not working,  nothing rendering at all
        BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
        for (int i = 1; i <= bands; i++) {
            Band band = dataset.GetRasterBand(i);
            int[] buffer = new int[width * height];
            band.ReadRaster(0, 0, width, height, buffer);
            if (i < bands)
                image.getRaster().setSamples(0, 0, width, height, i-1, buffer);
            else
                setAlphaBand(image, (byte) 125);
        }
        // 将BufferedImage转换为PNG格式的二进制数据
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ImageIO.write(image, "PNG", baos);
        return baos.toByteArray();
    }

    private static void setAlphaBand(BufferedImage image, byte alpha) {
        alpha %= 0xff;
        for (int cx = 0; cx < image.getWidth(); cx++) {
            for (int cy = 0; cy < image.getHeight(); cy++) {
                int color = image.getRGB(cx, cy);
                int mc = (alpha << 24) | 0x00ffffff;
                int newcolor = color & mc;
                image.setRGB(cx, cy, newcolor);
            }
        }
    }

[sendreams]

the best solution is Add VSIGetMemFileBuffer function for java binding

[sendreams]

the net binding code in gdal_csharp.i, should translate to java

%rename (GetMemFileBuffer) wrapper_VSIGetMemFileBuffer;

%typemap(cstype) (vsi_l_offset *pnDataLength) "out ulong";
%typemap(imtype) (vsi_l_offset *pnDataLength) "out ulong";
%apply (unsigned long long *OUTPUT) {(vsi_l_offset *pnDataLength)}
%typemap(cstype) (int bUnlinkAndSeize) "bool";
%typemap(csin) (int bUnlinkAndSeize) "$csinput ? 1 : 0";

%inline {
GByte* wrapper_VSIGetMemFileBuffer(const char *utf8_path, vsi_l_offset *pnDataLength, int bUnlinkAndSeize)
{
    return VSIGetMemFileBuffer(utf8_path, pnDataLength, bUnlinkAndSeize);
}
}

%clear (vsi_l_offset *pnDataLength);

[sendreams]

After trying several times, it still doesn't work. I'm giving up. I just want to improve the performance of exporting images; currently, it can only be done through disk I/O, which is too inefficient. @rouault

%module gdal

%{
    // 这里放置C++代码，SWIG需要知道这些C++的定义
    //typedef unsigned char GByte;

    typedef struct {
        GByte* buffer;  // 指向字节数组的指针
        long length;    // 数据长度
    } BufferWithLength;
%}

// 这里是告诉SWIG如何处理Java中的BufferWithLength类
%typemap(jstype) BufferWithLength "BufferWithLength";
%typemap(jout) BufferWithLength {
    jobject bufferObj = jenv->NewObject(BufferWithLength_class, BufferWithLength_ctor);
    jbyteArray byteArray = jenv->NewByteArray($1.length);
    jenv->SetByteArrayRegion(byteArray, 0, $1.length, (jbyte*)$1.buffer);
    jenv->SetObjectField(bufferObj, BufferWithLength_buffer, byteArray);
    jenv->SetLongField(bufferObj, BufferWithLength_length, (jlong)$1.length);
    delete[] $1.buffer;  // 释放 C++ 中的内存
    $result = bufferObj;
}

// 在Java中定义 BufferWithLength 类
%typemap(javacode) BufferWithLength %{
    public class BufferWithLength {
        public byte[] buffer;
        public long length;
        public BufferWithLength() {
            this.buffer = null;
            this.length = 0;
        }
    }
%}

// 包装C++的函数
%inline {
BufferWithLength wrapper_VSIGetMemFileBuffer(const char* pszFilename, int bUnlinkAndSeize)
{
    vsi_l_offset pnDataLength;
    GByte* buffer = VSIGetMemFileBuffer(pszFilename, &pnDataLength, bUnlinkAndSeize);

    BufferWithLength result;
    result.buffer = new GByte[pnDataLength];  // 为缓冲区分配内存
    memcpy(result.buffer, buffer, pnDataLength);  // 复制数据
    result.length = static_cast<long>(pnDataLength);  // 设置长度，确保类型匹配

    return result;  // 返回BufferWithLength结构体
}
}

[rouault]

Implemented in https://github.com/OSGeo/gdal/pull/11210

[sendreams]

@rouault thank you very very much.

%inline %{
GByte* wrapper_VSIGetMemFileBuffer(const char *utf8_path, vsi_l_offset *length)
{
    return VSIGetMemFileBuffer(utf8_path, length, 0);
}
%}

For the last argument of VSIGetMemFileBuffer, you are passing 0. Does this mean that after calling this function, the data remains in memory?

To test my assumption, I continue to call the function like this:

   @Test
    public void testGetMemFileBuffer()
    {
        gdal.FileFromMemBuffer("/vsimem/test", new byte[] {1, 2, 3});
        byte[] res = gdal.GetMemFileBuffer("/vsimem/test");
        res = gdal.GetMemFileBuffer("/vsimem/test");
        if (res.length != 3 )
            throw new RuntimeException("res.length != 3");
        if (res[0] != 1 || res[1] != 2 || res[2] != 3)
            throw new RuntimeException("res != {1, 2, 3}");
    }

However, calling it the second time causes the JVM to crash. I'm a bit confused here. Could you clarify this for me?

[rouault]

@sendreams Thanks for the testing. There was indeed a double-free issue. I've amended the pull request to fix that

[sendreams]

@rouault The VSI interface is a high-performance data operation interface with great potential for web applications. I hope more wrappers for it will be developed in the future. thanks again.

[sendreams]

If the VSI file is not released here, a VSI file release java interface would be necessary; otherwise, frequent calls could lead to a memory explosion.

[rouault]

a VSI file release java interface would be necessary

gdal.Unlink(filename). Cf https://github.com/rouault/gdal/blob/7baa26ebea9a7238c2edda198d261d3d98af688b/swig/java/javadoc.java#L248

[sendreams]

thanks, i got it.