Mat, Imgproc.resize Possible memory leak

[Racv]

Hi Folks,

I’m encountering a possible memory leak while using JavaCV for image resizing in a Spring-WebFlux application.

Environment Details:

• Kubernetes Pod: 8GB memory
• JVM Settings: -Xms=1024M, -Xmx=2048M
• Traffic: ~30 TPS on one pod

Issue: Memory utilization climbs to ~92% over a span of ~30 hours and then stabilises.

• I’ve ensured that all Mat objects are properly released and also employed PointerScope for memory management.
• JVM heap memory usage never exceeds 1GB.

Despite these efforts, memory usage continues to rise steadily over time. Below is the snippet of code used for resizing images:

try (PointerScope pointerScope = new PointerScope()) {
    Mat mat = Imgcodecs.imread(inputMediaPath.toString());
    Size size = new Size(width, height);
    Mat resizedMat = new Mat();
    Imgproc.resize(mat, resizedMat, size, 0, 0, Imgproc.INTER_AREA);

    Imgcodecs.imwrite(outputMediaPath.toString(), resizedMat);
    mat.release();
    resizedMat.release();
    // Tried both with and without pointerScope.deallocate(), but memory issues persist.
    pointerScope.deallocate();
}

Any advice on resolving this issue would be greatly appreciated. I’ve tried several approaches but cannot pinpoint the root cause of the rising memory usage.

Thanks in advance for your help!

Best regards, Ravi

[saudet]

Please try to set the "org.bytedeco.javacpp.nopointergc" system property to "true".

[Racv]

Still same behaviour.

I am using 1.5.10 version of javacv. and have also tried setting up these properties.

-Dorg.bytedeco.javacpp.maxPhysicalBytes=1G \
-Dorg.bytedeco.javacpp.maxBytes=512M \
-Dorg.bytedeco.javacpp.maxDeallocatorCache=5M \
-Dorg.bytedeco.javacpp.debug=true \

[saudet]

Please try to use the C++ API with JavaCPP instead of the Java API of OpenCV because the latter is not very well implemented.

[Racv]

Thanks for the input @saudet , I tried with simple javacpp API and also for easy debugging, created a small spring boot application with a single controller which only does resize, here also I am seeing similar behaviour. Pointer.physcialBytes() started with 1038M somewhere and now reaching ~1700M after continuous load testing with ~40 TPS for last 13hr.

package com.example.demo;

import jakarta.annotation.PostConstruct;
import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.Loader;
import org.bytedeco.javacpp.Pointer;
import org.bytedeco.opencv.global.opencv_imgcodecs;
import org.bytedeco.opencv.global.opencv_imgproc;
import org.bytedeco.opencv.opencv_core.Mat;
import org.bytedeco.opencv.opencv_core.Size;
import org.bytedeco.opencv.opencv_java;
import org.springframework.core.io.InputStreamResource;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;

import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Map;

@RestController
@RequestMapping("<BasePath>")
public class Controller {

    @PostConstruct
    void init() {
        Loader.load(opencv_java.class);
    }

    @GetMapping(value = "/{mediaTenant}/{mediaId}/{mediaName}")
    public ResponseEntity<StreamingResponseBody> processMedia(@RequestParam Map<String, String> params) throws IOException {
        String inputMediaPath = "media.png";
        Path outputMediaPath = Files.createTempFile("test", ".png");

        try (Mat mat = opencv_imgcodecs.imread(String.valueOf(inputMediaPath));
             Mat resizedMat = new Mat();
             BytePointer bytePointer = new BytePointer(String.valueOf(outputMediaPath))) {

            if (mat.empty()) {
                throw new RuntimeException("Could not read the input image.");
            }

            String newWidth = params.get("w");
            String newHeight = params.get("h");
            Size size = new Size(Integer.parseInt(newWidth), Integer.parseInt(newHeight));

            // Resize the image
            opencv_imgproc.resize(mat, resizedMat, size, 0D, 0D, opencv_imgproc.INTER_AREA);

            // Write the resized image to the output path
            opencv_imgcodecs.imwrite(bytePointer, resizedMat);

            // Stream the file as response and clean up after
            StreamingResponseBody responseBody = outputStream -> {
                try (InputStream fileStream = new FileInputStream(String.valueOf(outputMediaPath))) {
                    byte[] buffer = new byte[1024];
                    int bytesRead;
                    while ((bytesRead = fileStream.read(buffer)) != -1) {
                        outputStream.write(buffer, 0, bytesRead);
                    }
                    outputStream.flush();
                } finally {
                    // Clean up the temporary file
                    Files.deleteIfExists(outputMediaPath);
                }
            };

            mat.release();
            resizedMat.release();
            size.deallocate();
            bytePointer.deallocate();
            HttpHeaders headers = new HttpHeaders();
            headers.add(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + outputMediaPath.toFile().getName() + "\"");

            System.out.println(Pointer.physicalBytes()/ (1024*1024));
            return ResponseEntity.ok()
                    .headers(headers)
                    .contentType(MediaType.IMAGE_PNG)
                    .body(responseBody);

        }
    }
}

# Base image with JRE 21 from the private registry
FROM eclipse-temurin:21

RUN mkdir -p /path/to/image/folder

COPY media.png /path/to/image/folder

RUN mkdir -p /opt/dcxp-media-delivery-api /appl/media \
    && apt-get update -y \
        && apt install libjemalloc-dev -y \
         && apt-get install -y libgtk2.0-0 \ # This library is required by opencv.
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

COPY startup.sh /opt/app/startup.sh
RUN chmod +x /opt/app/startup.sh
COPY app.jar /opt/app/app.jar

EXPOSE 8080

# Set the working directory
WORKDIR /opt/app

CMD ["./startup.sh"]

• startup.sh

  
  #!/bin/bash

export MALLOC_CONF="prof:true,prof_leak:true,lg_prof_interval:30,lg_prof_sample:17,prof_prefix:/opt/app/prof/"

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so java -Xms1024M -Xmx2048M \ -Dorg.bytedeco.javacpp.maxPhysicalBytes=1G \ -XX:NativeMemoryTracking=detail \ -Dorg.bytedeco.javacpp.nopointergc=true \ -Dorg.bytedeco.javacpp.maxBytes=512M \ -Dorg.bytedeco.javacpp.debug=true \ -jar app.jar

docker run -p 8080:8080 --cpus="4" --memory="3500M" app:2

I have been tracking memory utilisation using `docker stats`, now it's showing 80% memory consumption.

Also I checked the JVM heap memory, it's not going beyond 1GB

[saudet]

Please try to use PointerScope: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

[Racv]

Hi @saudet,

I also tried using PointerScope and observed similar behaviour.

I adjusted the configuration slightly as follows:

java -Xms512M -Xmx1024M \
-Dorg.bytedeco.javacpp.maxBytes=1000M \
-Dorg.bytedeco.javacpp.maxPhysicalBytes=2000M \
-Dorg.bytedeco.javacpp.nopointergc=true \
-jar app.jar

For the Docker container:

• cpus: 6
• memory: 6000M

Here's the relevant Java code snippet:

try (PointerScope pointerScope = new PointerScope()) {
    Mat mat = opencv_imgcodecs.imread(String.valueOf(inputMediaPath));
    Mat resizedMat = new Mat();
    BytePointer bytePointer = new BytePointer(String.valueOf(outputMediaPath));
    if (mat.empty()) {
        throw new RuntimeException("Could not read the input image.");
    }

    String newWidth = params.get("w");
    String newHeight = params.get("h");
    Size size = new Size(Integer.parseInt(newWidth), Integer.parseInt(newHeight));

    // Resize the image
    opencv_imgproc.resize(mat, resizedMat, size, 0D, 0D, opencv_imgproc.INTER_AREA);

    // Write the resized image to the output path
    opencv_imgcodecs.imwrite(bytePointer, resizedMat);

    // Stream the file as a response and clean up afterward
    StreamingResponseBody responseBody = outputStream -> {
        try (InputStream fileStream = new FileInputStream(String.valueOf(outputMediaPath))) {
            byte[] buffer = new byte[1024];
            int bytesRead;
            while ((bytesRead = fileStream.read(buffer)) != -1) {
                outputStream.write(buffer, 0, bytesRead);
            }
            outputStream.flush();
        } finally {
            // Clean up the temporary file
            Files.deleteIfExists(outputMediaPath);
        }
    };

    mat.deallocate();
    resizedMat.deallocate();
    size.deallocate();
    bytePointer.deallocate();
    HttpHeaders headers = new HttpHeaders();
    headers.add(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + outputMediaPath.toFile().getName() + "\"");

    System.out.println(Pointer.physicalBytes() / (1024 * 1024));
    return ResponseEntity.ok()
            .headers(headers)
            .contentType(MediaType.IMAGE_PNG)
            .body(responseBody);
}

After 40 hours of testing with a load of 50 TPS, I noticed that Pointer.physicalBytes() never exceeded ~1800M, but the container memory usage still reached 96%.

I am not sure what is causing this memory leak. Since javacv provides all the features we need for our application, we are keen to stick with it, but this memory issue is becoming a significant blocker.

Please let me know if there's anything we can do to resolve this issue.

Thanks in advance, Ravi

[saudet]

After 40 hours of testing with a load of 50 TPS, I noticed that Pointer.physicalBytes() never exceeded ~1800M, but the container memory usage still reached 96%.

That just sounds like memory fragmentation. How are you sure this is even related to JavaCV?

[Racv]

Thanks @saudet for immediate reply,

I don't have in-depth knowledge on this topic, but what could be causing this memory fragmentation? The demo application only has one functionality, which is resizing images. Do you have any ideas on what might be causing this issue?

It's just a single spring controller for resizing.

[saudet]

Reallocating native memory a lot like that can cause memory fragmentation, but there's probably something else going on. If you could reproduce that outside Spring in a standalone application, this is something we could say might be related to JavaCV, but at this point, it could be anything really

[Racv]

Thanks @saudet , let me try that. Meanwhile I have pushed the demo code here https://github.com/Racv/javacv-demo.

[saudet]

Also please make sure to set the "org.bytedeco.javacpp.nopointergc" system property to "true".

[Racv]

Hi @saudet , I tried with standalone application as well, seeing similar behaviour

package org.example;

import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.Loader;
import org.bytedeco.javacpp.Pointer;
import org.bytedeco.javacpp.PointerScope;
import org.bytedeco.opencv.global.opencv_imgcodecs;
import org.bytedeco.opencv.global.opencv_imgproc;
import org.bytedeco.opencv.opencv_core.Mat;
import org.bytedeco.opencv.opencv_core.Size;
import org.bytedeco.opencv.opencv_java;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;

public class App {

    public void resize() throws IOException {
        String inputMediaPath = "media.png";
        Path outputMediaPath = Files.createTempFile("test", ".png");

        try (PointerScope pointerScope = new PointerScope()) {
            Mat mat = opencv_imgcodecs.imread(String.valueOf(inputMediaPath));
            Mat resizedMat = new Mat();
            BytePointer bytePointer = new BytePointer(String.valueOf(outputMediaPath));
            if (mat.empty()) {
                throw new RuntimeException("Could not read the input image.");
            }

            Random random = new Random();
            String newWidth = String.valueOf(random.nextInt(2001) + 1);
            String newHeight = String.valueOf(random.nextInt(2001) + 1);
            Size size = new Size(Integer.parseInt(newWidth), Integer.parseInt(newHeight));

            // Resize the image
            opencv_imgproc.resize(mat, resizedMat, size, 0D, 0D, opencv_imgproc.INTER_AREA);

            // Write the resized image to the output path
            opencv_imgcodecs.imwrite(bytePointer, resizedMat);

            mat.deallocate();
            resizedMat.deallocate();
            size.deallocate();
            bytePointer.deallocate();

            Files.deleteIfExists(outputMediaPath);
        }
    }

    public static void main(String[] args) throws IOException {
        Loader.load(opencv_java.class);
        int i = 0;
        int j=0;
        AtomicLong count = new AtomicLong();
        App app = new App();
        ExecutorService executor = new ThreadPoolExecutor(
                10,                       // Core pool size (number of threads)
                20,                       // Maximum pool size (limit number of threads)
                60L, TimeUnit.SECONDS,     // Time to keep idle threads alive
                new ArrayBlockingQueue<>(1000), // Bounded task queue of size 500
                new ThreadPoolExecutor.CallerRunsPolicy() // Handler when the queue is full
        );// Only allow 1000 tasks to be
        for(i=0; i < 100000 ; i++) {
            for(j=0; j<100000; j++) {
                executor.submit(() -> {
                    try {
                        app.resize();
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                    System.out.println(Pointer.physicalBytes()/ (1024*1024) + " : " + count.getAndIncrement());
                    // Calculate used memory
                    Runtime runtime = Runtime.getRuntime();
                    long usedMemory = (runtime.totalMemory() - runtime.freeMemory()) /(1024*1024);

                    // Total memory currently available to the JVM (committed memory)
                    long committedMemory = runtime.totalMemory() / (1024 * 1024);

                    // Maximum memory the JVM can use (based on the -Xmx setting)
                    long maxMemory = runtime.maxMemory() / (1024* 1024);
                    System.out.println("Used: "+usedMemory+", Commited: "+ committedMemory+", Max: "+ maxMemory);
                });
            }
        }
        executor.shutdown();
    }
}

java -Xms512M -Xmx1024M \
-Dorg.bytedeco.javacpp.maxBytes=1000M \
-Dorg.bytedeco.javacpp.maxPhysicalBytes=2000M \
-Dorg.bytedeco.javacpp.nopointergc=true \
-jar app.jar

after 60hr, memory reached 99% for 6GB container.

[saudet]

I see, and this happens within opencv_imgproc.resize()? If you comment out that line, then it doesn't leak?

[Racv]

yes @saudet

[saudet]

I see, that sounds like an issue with OpenCV itself 🤔

[Racv]

hmm.. from what I see currently javacv point to 4.9.0 version of opencv, is it possible to configure it to use latest version 4.10.0 ?

[saudet]

Yes, Please try again with the snapshots: http://bytedeco.org/builds/

[Racv]

getting this error and looks like it's still pointing to 4.9.0 of opencv. I am also using this plugin

plugins {
    // Apply the application plugin to add support for building a CLI application in Java.
    id 'org.bytedeco.gradle-javacpp-platform' version "1.5.10"
    id 'application'
    id 'java'
}

[saudet]

That's because I had not updated JavaCV itself. I've done that in commit c849f4794423205b7286c91ce38bf5431e208b19 so please try again!

[Racv]

Hi @saudet , I tried debugging this issue more and also opened up an issue in opencv official repo. But I noticed that this memory leak is not happening with just c++ code.

#include <opencv2/opencv.hpp>
#include <sys/resource.h>
#include <cstdio>
#include <cstdlib>              // For rand and srand
#include <ctime>                // For time to seed rand
#include <thread>               // For std::thread
#include <vector>               // For std::vector
#include <string>               // For std::string
#include <future>               // For async tasks

void print_memory_usage() {
    struct rusage usage;
    getrusage(RUSAGE_SELF, &usage);
    printf("Memory usage: %ld KB\n", usage.ru_maxrss);
}

void image_resize(const char* input_path, const char* output_path, int new_width, int new_height) {
    // Read the image from file
    cv::Mat input_image = cv::imread(input_path);
    if (input_image.empty()) {
        printf("Could not read the image: %s\n", input_path);
        return;
    }

    // Create an empty matrix to store the resized image
    cv::Mat resized_image;
    // Resize the image
    cv::resize(input_image, resized_image, cv::Size(new_width, new_height));

    // Write the resized image to file
    //cv::imwrite(output_path, resized_image);

    resized_image.release();
    input_image.release();
}

int main(int argc, char** argv)
{
    printf("OpenCV version: %d.%d.%d\n", CV_VERSION_MAJOR, CV_VERSION_MINOR, CV_VERSION_REVISION);
    print_memory_usage();
    int dimensions[] = {64, 128, 256, 512, 1024};

    for (int i = 1; i < 1000000; i++)
    {
        std::vector<std::future<void>> futures;
        for(int j = 1 ; j< 20; j++) {

            int width = dimensions[rand() % 4];
            int height = dimensions[rand() % 4];
            futures.push_back(std::async(std::launch::async, image_resize, "/app/media.png", "output.png", width, height));
            printf("count: %d\n", i * j);
            if(((j) % 10) == 0)
                print_memory_usage();

        }
        // Wait for all tasks to complete
            for (auto& fut : futures) {
                fut.get();
            }
            futures.clear();
    }

   // image_resize("swatch.jpeg", "output.png", dimensions[rand() % 4], dimensions[rand() % 4];);
    return 0;
}

But when I try to run same code using JNI, I see the memory leak is happening. I have tested with javacv and with direct c++ code as well.

#include <jni.h>
#include <opencv2/opencv.hpp>
#include <iostream>
#include <string>
#include <sys/resource.h>
#include <cstdio>

void print_memory_usage() {
    struct rusage usage;
    getrusage(RUSAGE_SELF, &usage);
    printf("Memory usage: %ld KB\n", usage.ru_maxrss);
}

extern "C" {

// JNI method implementation
JNIEXPORT void JNICALL Java_org_example_JniImageResizer_imageResize(JNIEnv* env, jobject, jstring inputPath, jstring outputPath, jint newWidth, jint newHeight) {
    // Convert Java strings to C++ strings
    printf("OpenCV version: %d.%d.%d\n", CV_VERSION_MAJOR, CV_VERSION_MINOR, CV_VERSION_REVISION);
    const char* inputPathChars = env->GetStringUTFChars(inputPath, nullptr);
    const char* outputPathChars = env->GetStringUTFChars(outputPath, nullptr);

    try {
        // Read the image from file
        cv::Mat input_image = cv::imread(inputPathChars);
        if (input_image.empty()) {
            std::cerr << "Could not read the image: " << inputPathChars << std::endl;
            return;
        }

        // Create an empty matrix to store the resized image
        cv::Mat resized_image;
        // Resize the image
        cv::resize(input_image, resized_image, cv::Size(newWidth, newHeight));

        // Write the resized image to the output file
        cv::imwrite(outputPathChars, resized_image);

        // Release the images
        input_image.release();
        resized_image.release();
    } catch (const cv::Exception& e) {
        std::cerr << "OpenCV Error: " << e.what() << std::endl;
    } catch (const std::exception& e) {
        std::cerr << "Standard Error: " << e.what() << std::endl;
    }

    // Release resources allocated for the Java strings
    env->ReleaseStringUTFChars(inputPath, inputPathChars);
    env->ReleaseStringUTFChars(outputPath, outputPathChars);
    print_memory_usage();
}

}  // extern "C"

package org.example;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;

public class JniImageResizer {

    // Native method declaration
    private native void imageResize(String inputPath, String outputPath, int newWidth, int newHeight);

    static {
        // Load the native shared library (libimage_resize.so)
        System.load("/home/azureuser/opencv/opencv-so/libs/libimage_resize.so");
    }

    public void resizeImage() {
        try {
            long start = System.currentTimeMillis();
            Path outputMediaPath = Files.createTempFile("test", ".png");

            // Generating random dimensions
            Integer[] h = {300, 450};
            Integer[] w = {400, 500};
            Random random = new Random();
            int newWidth = h[random.nextInt(2)];
            int newHeight = w[random.nextInt(2)];

            // Call the native method
            imageResize("/app/media.png",
                    outputMediaPath.toString(), newWidth, newHeight);

            // Clean up
            Files.deleteIfExists(outputMediaPath);
            long end = System.currentTimeMillis();
            System.out.println("time: " + (end - start));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        System.out.println(System.getProperty("java.library.path"));
        System.out.println("loaded");

        AtomicLong count = new AtomicLong();

        ExecutorService executor = new ThreadPoolExecutor(
                15,                       // Core pool size
                20,                       // Maximum pool size
                60L, TimeUnit.SECONDS,     // Keep-alive time
                new ArrayBlockingQueue<>(1000), // Task queue size
                new ThreadPoolExecutor.CallerRunsPolicy() // Handler when the queue is full
        );

        JniImageResizer app = new JniImageResizer();

        for (int i = 0; i < 10000; i++) {
            for (int j = 0; j < 10000; j++) {
                executor.submit(() -> {
                    try {
                        app.resizeImage();
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                    System.out.println("Count: " + count.getAndIncrement());

                    // Calculate used memory
                    Runtime runtime = Runtime.getRuntime();
                    long usedMemory = (runtime.totalMemory() - runtime.freeMemory()) / (1024 * 1024);
                    long committedMemory = runtime.totalMemory() / (1024 * 1024);
                    long maxMemory = runtime.maxMemory() / (1024 * 1024);
                    System.out.println("Used: " + usedMemory + " MB, Committed: " + committedMemory + " MB, Max: " + maxMemory + " MB");
                });
            }
        }
        executor.shutdown();
    }
}

Do you see any issue in JNI version of resize code?

[saudet]

Please try in Java without threads. It's entirely possible there is a threading bug in OpenCV.