JavaFXFrameConverter is using way too much memory in a specific case.

[iexavl]

(I posted this question on stackoverflow but I assumed it would be better If I opened an issue here as well) The JavaFXFrameConvertervconvert() method is consuming an outrageous amount of memory. To elaborate a bit more, it does not happen usually. If I just make an instance of the class in my main method, grab a frame via FFMpegFrameGrabber and give it to the convert() method the memory usage is pretty much none. However, when I attempt to do pretty much the exact same in a class I made using an ExecutorService my memory usage jumps up to 8 gigabytes when convert is called. The converter and executor service are declared as member variables of my class. Namely:

    final JavaFXFrameConverter converter = new JavaFXFrameConverter();
    private  ExecutorService           videoExecutor;

(the videoExecutor is instantiated in the constructor of the class:

videoExecutor=Executors.newSingleThreadExecutor();

Now, the method I am using for processing of the video frames is this:

    private void processVideo(){
        videoExecutor.submit(() -> {
            processingVideo.set(true);
            try{

                while(processingVideo.get() && (videoQueue.peek())!=null){

                    final Frame cloneFrame = videoQueue.poll();
                    final Image image = converter.convert(cloneFrame);
                    final long timeStampDeltaMicros = cloneFrame.timestamp - timer.elapsedMicros();
                    if (timeStampDeltaMicros > 0) {
                        final long delayMillis = timeStampDeltaMicros / 1000L;
                        try {
                            Thread.sleep(delayMillis);
                        } catch (InterruptedException e) {
                            Thread.currentThread().interrupt();
                        }
                    }

                    cloneFrame.close();
                    System.out.println("submitted image");
                videoListener.submitData(image);
                }
            }catch (NullPointerException e){
                NullPointerException ex = new NullPointerException("Error while processing video frames.");
            videoListener.notifyFailure(ex);
            }

            processingVideo.set(false);

        });
    }

I sent the whole method for a bit more context but realistically speaking the only part that is of real significance is the converter.conver(cloneFrame); I used the intelliJ profiler and also the debugger and this is exactly where the problem occurs. When convert is called after doing some stuff it eventually ends up in this method:

        public <T extends Buffer> void getPixels(int x, int y, int w, int h, WritablePixelFormat<T> pixelformat, T buffer, int scanlineStride) {
            int fss = this.frame.imageStride;
            if (this.frame.imageChannels != 3) {
                throw new UnsupportedOperationException("We only support frames with imageChannels = 3 (BGR)");
            } else if (!(buffer instanceof ByteBuffer)) {
                throw new UnsupportedOperationException("We only support bytebuffers at the moment");
            } else {
                ByteBuffer bb = (ByteBuffer)buffer;
                ByteBuffer b = (ByteBuffer)this.frame.image[0];

                for(int i = y; i < y + h; ++i) {
                    for(int j = x; j < x + w; ++j) {
                        int base = 3 * j;
                        bb.put(b.get(fss * i + base));
                        bb.put(b.get(fss * i + base + 1));
                        bb.put(b.get(fss * i + base + 2));
                        bb.put((byte)-1);
                    }
                }

            }
        }

Now, everything up until this point is fine. The memory usage is at around 130mb but alas, when execution enters in these 2 for loops that's where the downright stupid memory usage occurs. Every single one of these bb.put calls is netting me around 3 more megabytes of memory usage. By the end of it you can probably guess what happens. Also all of these memory allocations do happen on the stack so I'm assuming that's why my memory usage stops at around 8-8.5 gigabytes otherwise the program would crash (that has also happened, out of memory exception thrown, but it doesn't usually happen, it kind of just lingers at those 8 gigabytes.) Frankly speaking I'm at a bit of a loss. I haven't seen virtually anyone anywhere mention this ever and I ran out of things to try to fix this so I am making this post.

By the way another thing I tried is make the ExecutorService in the same class as the main method and when I submitted there I also didn't have these memory problems. if there is any additional information I can provide I would be happy to do so.

[saudet]

@johanvos Any ideas what's going on here?

[iexavl]

@saudet okay I am getting more confused by the minute. There might be something I don't know that's going on maybe something related to the debugger (I damn hope) but I haven't seen this happen before ever so I'm assuming not? If I try stepping in one of the lines like bb.put(b.get(fss * i + base)); for example I go into the get() method and it looks like this:

    public byte get(int i) {
        try {
            return ((SCOPED_MEMORY_ACCESS.getByte(session(), null, ix(checkIndex(i)))));
        } finally {
            Reference.reachabilityFence(this);
        }
    }

now you see the method session() or ix()? Yeah upon calling either of them (and I mean JUST calling, nothing has happened in the actual method yet. I step into it and boom instant 2-3 megabytes)and exiting either of them I am netting myself an additional 2-3 megabytes per each operation for around 12 megabytes in total. Now I can see why my memory shoots up to 8 gigabytes in the matter of literal seconds but what the hell is going on?

[saudet]

BTW, you might want to check out this example from @rladstaetter, which uses a faster API available in recent versions of JavaFX anyway: https://github.com/rladstaetter/javacv-webcam