artclarke / xuggle-xuggler

Xuggle's Xuggler Java API for Video -- DEPRECATED
GNU Lesser General Public License v3.0
360 stars 226 forks source link

Massive memory leak in ArgbConverter.java#toImage #4

Closed naftulikay closed 11 years ago

naftulikay commented 12 years ago

I'm writing an application which needs to decode transparent frames from an FLV file, so I've activated the scary ArgbConverter in order to make this happen. Nothing about the actual output images would indicate any problem with FFMPEG support, but I noticed a massive memory leak in the method. All of the bytes of each individual frame are kept in memory indefinitely, so if I decode a short, 4 second video into a sequence of images, the actual allocated memory jumps to around 150MB, while the virtual memory jumps to over 700MB:

http://i.imgur.com/nJISz.png

I'm even calling IVideoConverter.delete() after each frame is generated to try and wipe it, but this is definitely an in-JVM bug and not something that directly has to do with C types, as far as I can see.

The code which seems to be causing the leak is here:

ArgbConverter.java:181-201

  final ByteBuffer byteBuf = picture.getByteBuffer(ref);

  // now, for this class of problems, we don't want the code
  // to switch byte order, so we'll pretend it's in native java order

  byteBuf.order(ByteOrder.BIG_ENDIAN);
  final IntBuffer intBuf = byteBuf.asIntBuffer();
  final int[] ints = new int[picture.getSize() / 4];
  intBuf.get(ints, 0, ints.length);

  // create the data buffer from the ints

  final DataBufferInt db = new DataBufferInt(ints, ints.length);

  // create an a sample model which matches the byte layout of the
  // image data and raster which contains the data which now can be
  // properly interpreted

  final SampleModel sm = new SinglePixelPackedSampleModel(db.getDataType(),
      w, h, mBitMasks);
  final WritableRaster wr = Raster.createWritableRaster(sm, db, null);

None of these items are ever unallocated, the method simply returns the generated BufferedImage instance shortly afterward.

I'm not sure what the best way to clear out these buffers and int[]s would be, so I'm open to suggestions from those who know the codebase better (ahem, @xuggle), but yeah, that's the issue.

artclarke commented 12 years ago

I don't think you're right. See Line 207 where the items you claim are not deleted are actually deleted. That's the xuggle objects at least.

The other (Java) objectsI cannot be responsible for -- the java GC should collect them (e.g. the int[] and the RasterModel). I suspect they are (as you state) staying in memory, but not because they are 'leaked'; only because they are not collected. To see if I'm right, force a garbage collection and you'll likely see that memory drops back down. If not, use something like YourKit to find where the leak is :) Alternately, if there is a cleanup call for those Java objects, please let me know what they are.

That said, I think that method in general is not strong, and you may want to reimplement the same method in a way that COPIES that data using the IBuffer commands into an array that you maintain separately. That way, you don't allocate new objects on the heap repeatedly and don't stress the collector as much.

On Tue, Jul 31, 2012 at 1:15 PM, rfkrocktk < reply@reply.github.com

wrote:

I'm writing an application which needs to decode transparent frames from an FLV file, so I've activated the scary ArgbConverter in order to make this happen. Nothing about the actual output images would indicate any problem with FFMPEG support, but I noticed a massive memory leak in the method. All of the bytes of each individual frame are kept in memory indefinitely, so if I decode a short, 4 second video into a sequence of images, the actual allocated memory jumps to around 150MB, while the virtual memory jumps to over 700MB:

http://i.imgur.com/nJISz.png

I'm even calling IVideoConverter.delete() after each frame is generated to try and wipe it, but this is definitely an in-JVM bug and not something that directly has to do with C types, as far as I can see.

The code which seems to be causing the leak is here:

ArgbConverter.java:181-201

  final ByteBuffer byteBuf = picture.getByteBuffer(ref);

  // now, for this class of problems, we don't want the code
  // to switch byte order, so we'll pretend it's in native java order

  byteBuf.order(ByteOrder.BIG_ENDIAN);
  final IntBuffer intBuf = byteBuf.asIntBuffer();
  final int[] ints = new int[picture.getSize() / 4];
  intBuf.get(ints, 0, ints.length);

  // create the data buffer from the ints

  final DataBufferInt db = new DataBufferInt(ints, ints.length);

  // create an a sample model which matches the byte layout of the
  // image data and raster which contains the data which now can be
  // properly interpreted

  final SampleModel sm = new

SinglePixelPackedSampleModel(db.getDataType(), w, h, mBitMasks); final WritableRaster wr = Raster.createWritableRaster(sm, db, null);

None of these items are ever unallocated, the method simply returns the generated BufferedImage instance shortly afterward.

I'm not sure what the best way to clear out these buffers and int[]s would be, so I'm open to suggestions from those who know the codebase better (ahem, @xuggle), but yeah, that's the issue.


Reply to this email directly or view it on GitHub: https://github.com/xuggle/xuggle-xuggler/issues/4

http://www.xuggle.com/ xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and video.

Use Xuggle to get the power of FFmpeg in Java.

RyanfaeScotland commented 11 years ago

It doesn't look like this is related soley / directly to ArgbConverter.java. I'm having a similar problem which appears to stem from BgrConverter.java#toImage. ( Image of profiler: http://www.mybrillgamesite.com/miscImages/mem.png ) I built your snapshot rfkrocktk but (unsurprisingly) it didn't fix my issue.

Think I'll try apply the same changes to BgrConverter and see if that does the job.

naftulikay commented 11 years ago

Keep us updated, interested in what you find.

Thanks,

On Thu, Dec 6, 2012 at 6:49 AM, RyanfaeScotland notifications@github.comwrote:

It doesn't look like this is related soley / directly to ArgbConverter.java. I'm having a similar problem which appears to stem from BgrConverter.java#toImage. ( Image of profiler: http://www.mybrillgamesite.com/miscImages/mem.png ) I built your snapshot rfkrocktk but (unsurprisingly) it didn't fix my issue.

Think I'll try apply the same changes to BgrConverter and see if that does the job.

— Reply to this email directly or view it on GitHubhttps://github.com/xuggle/xuggle-xuggler/issues/4#issuecomment-11088098.

RyanfaeScotland commented 11 years ago

Those changes made no difference either. Investigations continue :)

Sadly we don't have too much time to dedicate to this so may end up moving to different method of decoding if we can't resolve it soonish but I'll keep looking for today at least.

Just for background we are decoding a H.264 stream into BufferedImages.

RyanfaeScotland commented 11 years ago

What's the etiquette here, edit previous comment with new information of post a fresh?

Posting a fresh till I know.

So I'm not convinced my problem lies with the above, or in fact there is definitely a problem with the above. Certainly the routines use a lot of memory but I'm not convinced they are leaking it. I think my problem is that I don't have enough memory to run Netbeans + Profiler + My Application without the whole thing running out of memory just due to normal processing. To test I ran the system without the IDE and profiling and just checked the output from 'free -m' occasionally and I found it did balance out after a while (at about the 1 Gig mark). However this was with the 'fixed' version that TK produced (and I added to).

To test I've set the system up on our test bed with Xuggler 5.4 and am going to run it over the weekend with 'free -m -s 30' outputting to a log. I also have several other systems running with the altered version so I'll see how they cope come Monday as well. Will let you know the result next week.

bokken commented 11 years ago

What JRE are you using? Is the result of final ByteBuffer byteBuf = picture.getByteBuffer(ref);

A Direct ByteBuffer? I believe that the sun jre (at least in 1.6) makes some attempt to pool the direct ByteBuffers. It is possible to cast to a sun specific class and then call sun specific methods to force the memory to be reclaimed.

artclarke commented 11 years ago

Also, you can force xuggler to delete the underlying native buffer by using the ref you pass in.

Sent from my iPhone

On Dec 8, 2012, at 11:47 AM, bokken notifications@github.com wrote:

What JRE are you using? Is the result of final ByteBuffer byteBuf = picture.getByteBuffer(ref);

A Direct ByteBuffer? I believe that the sun jre (at least in 1.6) makes some attempt to pool the direct ByteBuffers. It is possible to cast to a sun specific class and then call sun specific methods to force the memory to be reclaimed.

— Reply to this email directly or view it on GitHub.

RyanfaeScotland commented 11 years ago

I think I've been a bit daft and mis-interpreted the chart. My thinking is now that the 'Bytes Allocated' figure I was looking at was an OVERALL figure and not the memory being used at that moment it time. It grew to 14 Gig at one point which sort of tipped me off.

The live results shows it is getting big (70M is the most I saw it) but it is being cleared again so I'm starting to think our problem isn't related to this. I'll return if I change my mind or find something related.

I'm JRE 1.7 to answer your question, haven't tried the byteBuf check.

artclarke commented 11 years ago

fixed intrunk.

RyanfaeScotland commented 11 years ago

Hey Art, welcome back. Good to see you on the go with Xuggler again (maybe you never left, been a while since I was here myself ;) )

artclarke commented 11 years ago

Just here closing down the shop actually and letting folks know where things are. Xuggle code-development has stopped. It is possible that I will announce a new project that takes up where Xuggle left off, but that's in the future :)

On Mon, Jul 22, 2013 at 8:21 AM, RyanfaeScotland notifications@github.comwrote:

Hey Art, welcome back. Good to see you on the go with Xuggler again (maybe you never left, been a while since I was here myself ;) )

— Reply to this email directly or view it on GitHubhttps://github.com/artclarke/xuggle-xuggler/issues/4#issuecomment-21351533 .

RyanfaeScotland commented 11 years ago

Well thanks for all the work, it's been a great help to us (and the source of much frustration of course!) and best of luck with your future endeavours whatever they may be.

jkolobok commented 11 years ago

How far is this future from present?