FFmpegFrameRecorder and FFmpegFrameGrabber may lead to memory leak

[z384647062]

I used FFmpegFrameRecorder and FFmpegFrameGrabber to convert audio data in mp3 format to PCM format. The following is my code and it has done well.

`

    public static void convert(String inputFile, String outputFile, int audioCodec, int sampleRate, int audioBitrate,
                           int audioChannels) {
    Frame audioSamples = null;

    FFmpegFrameRecorder recorder = null;

    FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputFile);

    if (start(grabber)) {
        recorder = new FFmpegFrameRecorder(outputFile, audioChannels);
        recorder.setAudioOption("crf", "0");
        recorder.setAudioCodec(audioCodec);
        recorder.setAudioBitrate(audioBitrate);
        recorder.setAudioChannels(audioChannels);
        recorder.setSampleRate(sampleRate);
        recorder.setAudioQuality(0);
        recorder.setAudioOption("aq", "10");

        if (start(recorder)) {
            try {
                while ((audioSamples = grabber.grab()) != null) {
                    recorder.setTimestamp(grabber.getTimestamp());
                    recorder.record(audioSamples);
                }

            } catch (FrameGrabber.Exception e1) {
            } catch (Exception e) {
            }
            stop(grabber);
            stop(recorder);
        }
    }

}

public static boolean start(FrameGrabber grabber) {
    try {
        grabber.start();
        return true;
    } catch (FrameGrabber.Exception e2) {
        try {
            grabber.restart();
            return true;
        } catch (FrameGrabber.Exception e) {
            try {
                grabber.stop();
            } catch (FrameGrabber.Exception e1) {
            }
        }

    }
    return false;
}

public static boolean start(FrameRecorder recorder) {
    try {
        recorder.start();
        return true;
    } catch (Exception e2) {
        try {
            recorder.stop();
            recorder.start();
            return true;
        } catch (Exception e) {
            try {
                recorder.stop();
            } catch (Exception e1) {
            }
        }
    }
    return false;
}

public static boolean stop(FrameGrabber grabber) {
    try {
        grabber.flush();
        grabber.stop();
        return true;
    } catch (FrameGrabber.Exception e) {
        return false;
    } finally {
        try {
            grabber.stop();
        } catch (FrameGrabber.Exception e) {
        }
    }
}

public static boolean stop(FrameRecorder recorder) {
    try {
        recorder.stop();
        recorder.release();
        return true;
    } catch (Exception e) {
        return false;
    } finally {
        try {
            recorder.stop();
        } catch (Exception e) {
        }
    }
}

`

For example, it can be used by

convert("test.mp3","test.pcm", avcodec.AV_CODEC_ID_PCM_S16LE, 16000, 16, 1);

But when I run these code(server) for a long time, the memory usage has reached to 9GB and keep increasing unless I restart the server.

[saudet]

Duplicate of #1105. Please upgrade to JavaCV 1.5.

[z384647062]

Duplicate of #1105. Please upgrade to JavaCV 1.5.

The version of JavaCV I used is 1.5.

`

org.bytedeco

        <artifactId>javacv-platform</artifactId>
        <version>1.5</version>
    </dependency>`

[saudet]

Ok, would it be possible to run your application with AddressSanitizer or Valgrind to see where the leak is occurring?

[duck123456]

Ok, would it be possible to run your application with AddressSanitizer or Valgrind to see where the leak is occurring?

My program has the same problem，memory leak！use valgrind --tool=memcheck --leak-check=full java -jar video-capture-server-1.0.0.jar ，It's very slow，so use jmap and pmap see the mem, java use mem :426469064 (byte) and process RSS :38156148 (kb) .

[saudet]

Ok, could you give me the smallest possible program that triggers this leak?

[duck123456]

Ok, could you give me the smallest possible program that triggers this leak?

public class TestGrabber {

@Test
public void start(){
    for(int i=0;i<10;i++){
        RunTest runTest=new RunTest();
        Thread thread=new Thread(runTest);
        thread.start();
    }
    while (true){

        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

public void test() { FFmpegFrameGrabber grabber =null; FFmpegFrameRecorderPlusV3 recorder = null; ByteArrayOutputStream outputStream = null; String requestStreamUrl = "D:\video_file\0c1666e1-86f2-4882-b68c-d45a5b59c6e8\0c1666e1-86f2-4882-b68c-d45a5b59c6e8-4.ts"; while (true) { try { grabber = new FFmpegFrameGrabber(requestStreamUrl); WeakReference weakVideoFrameGrabberV3 = new WeakReference(grabber); grabber.start(); outputStream = new ByteArrayOutputStream(); int len = grabber.getLengthInFrames(); recorder = new FFmpegFrameRecorderPlusV3(outputStream, UUIDUtil.getUUIdKey() + ".ts", grabber.getImageWidth(), grabber.getImageHeight(), 0); recorder.start(grabber.getFormatContext()); int index = 0; while (true) { recorder.recordPacket(grabber.grabPacket()); index++; if (index == len) { index = 0; try { outputStream.close(); } catch (IOException e) { e.printStackTrace(); } outputStream = new ByteArrayOutputStream(); recorder.resetOutStream(outputStream); break; } }

       } catch (FrameGrabber.Exception e) {
           // e.printStackTrace();
       } catch (FrameRecorder.Exception e) {
           e.printStackTrace();
       } finally {
           try {
               grabber.stop();
               grabber.release();
               grabber = null;
           } catch (FrameGrabber.Exception e) {
               e.printStackTrace();
           }
           if (recorder != null) {
               try {
                   recorder.stop();
                   recorder.release();
                   recorder = null;

               } catch (FrameRecorder.Exception e) {
                   e.printStackTrace();
               }
           }
           System.gc();
       }
   }

}

class RunTest implements  Runnable{

    @Override
    public void run() {
        test();
    }
}

} you can try it,but i don't sure triggers this leak, maybe not javacv leak，Because it is growing very slowly, I don't know how to see the effect quickly!

[saudet]

Thanks, but that's pretty long. Can you shorten it?

[duck123456]

Thanks, but that's pretty long. Can you shorten it?

sorry ！I don't know how to use memory leak detection tools, so I'm not sure what caused the leak. Can you provide some methods of screening? Let me quickly locate the source of the leak

[saudet]

Well, Valgrind can do that, but if you could reproduce the issue in a few lines of code, it would help to narrow down the search.

[saudet]

Can you do it without an OutputStream?

[duck123456]

Can you do it without an OutputStream?

ok，I try it ,but i need OutputStream to implement video segmentation

[duck123456]

Can you do it without an OutputStream? valgrind log: ==146226== LEAK SUMMARY: ==146226== definitely lost: 11,847 bytes in 529 blocks ==146226== indirectly lost: 4,762,087 bytes in 727 blocks ==146226== possibly lost: 36,444,407 bytes in 705 blocks ==146226== still reachable: 81,162,199 bytes in 10,291 blocks ==146226== of which reachable via heuristic: ==146226== newarray : 22,536 bytes in 1 blocks ==146226== suppressed: 0 bytes in 0 blocks ==146226== Reachable blocks (those to which a pointer was found) are not shown. ==146226== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==146226== ==146226== Use --track-origins=yes to see where uninitialised values come from ==146226== For lists of detected and suppressed errors, rerun with: -s

code: ` public static void main(String[] args) { String requestStreamUrl = "/home/developer/cbvsp_back_end/video-capture-server/video/0c1666e1-86f2-4882-b68c-d45a5b59c6e8-3.ts"; FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(requestStreamUrl); try {

        grabber.start();

    int len = grabber.getLengthInFrames();
        System.out.println("grabber getLengthInFrames"+len);

        for(int i=0;i<len;i++){

            grabber.grabPacket();

        }
    } catch (FrameGrabber.Exception e) {
        System.out.println("grabber  error");
        e.printStackTrace();
    }finally {
        try {
            grabber.stop();
            grabber.release();
        } catch (FrameGrabber.Exception e) {
            e.printStackTrace();
        }
    }
}`

[duck123456]

Do I need to provide detailed log files？

[saudet]

No, just isolating where the problem occurs is probably enough. Does it happen even without grabPacket()? Does it happen with grab()?

[duck123456]

No, just isolating where the problem occurs is probably enough. Does it happen even without grabPacket()? Does it happen with grab()?

just do gabber->start->grabPacket->stop ,no use grab() method

[duck123456]

No, just isolating where the problem occurs is probably enough. Does it happen even without grabPacket()? Does it happen with grab()?

definitely lost: 11,847 bytes in 529 blocks .Is this data normal?

[saudet]

No, but if it happens only with grabPacket(), and not with grab(), then it's only a problem with grabPacket().

[saudet]

Also, could you compare with an empty main() method? It might just be the JVM leaking memory.

[duck123456]

Also, could you compare with an empty main() method? It might just be the JVM leaking memory.

empty main() method log: ==148555== LEAK SUMMARY: ==148555== definitely lost: 1,719 bytes in 26 blocks ==148555== indirectly lost: 6,944 bytes in 15 blocks ==148555== possibly lost: 36,137,010 bytes in 649 blocks ==148555== still reachable: 79,197,813 bytes in 2,100 blocks ==148555== of which reachable via heuristic: ==148555== newarray : 22,536 bytes in 1 blocks ==148555== suppressed: 0 bytes in 0 blocks

[duck123456]

Also, could you compare with an empty main() method? It might just be the JVM leaking memory.

so,no only JVM leaking memory ,Perhaps these leaks can be ignored

[duck123456]

i see jvm dump memory 400M，why process memory 10G? Where is all that memory allocated？

[saudet]

Yeah, this seems to be an issue with grabPacket(). I'm unable to reproduce this with grab(). grabPacket() uses deprecated function calls though, so we should change that first and see how well that works, see issue #818.

[saudet]

@z384647062 If you do not like your process taking 9~10 GB of memory, make sure to set the maximum amount of memory your process can use to a lower value, by using java -Xmx option on the command line, for example.

[duck123456]

@z384647062 If you do not like your process taking 9~10 GB of memory, make sure to set the maximum amount of memory your process can use to a lower value, by using java -Xmx option on the command line, for example.

ok，Thank you very mach

[saudet]

We can also use PointerScope for tighter memory management. See issue https://github.com/bytedeco/javacv/issues/911#issuecomment-421444441 for an example with FFmpegFrameGrabber. We can do the same with FFmpegFrameRecorder.

[duck123456]

We can also use PointerScope for tighter memory management. See issue #911 (comment) for an example with FFmpegFrameGrabber. We can do the same with FFmpegFrameRecorder.

ok，thanks

[duck123456]

We can also use PointerScope for tighter memory management. See issue #911 (comment) for an example with FFmpegFrameGrabber. We can do the same with FFmpegFrameRecorder.

hi, I try to use PointerScope scope = new PointerScope() but releaseUnsafe() >pkt2.size() error occurred log: java.lang.NullPointerException: This pointer address is NULL. at org.bytedeco.ffmpeg.avcodec.AVPacket.size(Native Method)

https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L162-L164

[saudet]

You'll need to make sure stop() has been called before exiting the scope.

[duck123456]

You'll need to make sure stop() has been called before exiting the scope.

really! thanks

[saudet]

I think I've fixed all the memory leaks occurring in FFmpegFrameGrabber and FFmpegFrameRecorder. Please give it a try with the snapshots: http://bytedeco.org/builds/

[Arman06061999]

guys, please tell me what I'm doing wrong, I'm trying to convert AVI, MOV, WMV these files to mp4 format, point to the errors, the user from the web application will convert his video file to mp4. public File convertToMp4(File file) throws IOException, FrameGrabber.Exception, FrameRecorder.Exception { FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(file); grabber.start(); FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(file, 0); recorder.setFormat("mp4"); recorder.setFrameRate(grabber.getFrameRate()); recorder.setSampleRate(grabber.getSampleRate()); recorder.setVideoCodec(grabber.getVideoCodec()); recorder.setAudioCodec(grabber.getAudioCodec()); recorder.start(); Frame frame; while ((frame = grabber.grabFrame()) != null) { recorder.record(frame); } recorder.stop(); recorder.release(); grabber.stop(); grabber.release();

[saudet]

@Arman06061999 We can easily accomplish this with the ffmpeg program: http://bytedeco.org/javacpp-presets/ffmpeg/apidocs/org/bytedeco/ffmpeg/ffmpeg.html