search

bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.42k stars 1.57k forks source link

Rendering 4k frames with javacv #2077

Open iexavl opened 11 months ago

iexavl commented 11 months ago

So I am trying to render a 4k video with CanvasFrame . It appears it's kind of struggling with that task. I have this put in the VM options: -Dsun.java2d.opengl=true but it still seems to struggle rendering the frames. Everything is fragmented when the frames are displayed. Is there some way to fix this or is CanvasFrame just not fast enough for this task?

saudet commented 11 months ago

Please check out this example from @rladstaetter, which uses a faster API available in recent versions of JavaFX: https://github.com/rladstaetter/javacv-webcam

iexavl commented 11 months ago

@saudet does it matter if I use FFMpegFrameGrabber or OpenCVFrameGrabber?

iexavl commented 11 months ago

okay I did it and it still looks like crap. I'm starting to think the issue is somewhere else edit: I'm not sure at this point. I made a static variable and I am incrementing it with each submitted image. The video is 60 fps and its fine for about a second. After that the frame rate drops to like 4 fps.

iexavl commented 10 months ago

@saudet so I left this problem for a while because it isn't super important for my application but its definitely bugging me. I found out the problem is not in ImageView or CanvasFrame or anything related to actually displaying the image. The image itself is corrupted. This is what the image looks like originally: image And this is what it looks like after it has been decoded and processed via FFmpegFrameGrabber's processImage() method image

I am not sure what exactly is going wrong here but I somewhat doubt the decoding going wrong as I have ran into some decoding issues before and they usually result in visual glitches but for one it doesn't cause a color change and the decoder also cries out and I can see the error messages flooding in. This definitely looks more like some sort of format conversion bug. I am suspecting this :

  sws_scale(img_convert_ctx, new PointerPointer(picture), picture.linesize(), 0,
                        video_c.height(), new PointerPointer(picture_rgb), picture_rgb.linesize());
                frame.imageStride = picture_rgb.linesize(0);
                frame.image = image_buf;
                frame.opaque = picture_rgb;

but unfortunately I am not really all that versed in image conversion. Another thing to note is that this seems to actually happen with images bigger than the native resolution of my monitor. I tried it with 1440p and 4k, my monitor is 1080p, on two different videos and pretty much the same problem occurred. Take that with a grain of salt though because I tried changing the resolution of my monitor to 800x600p and when I played the exact same video that I played in 1440p in 720p (two versions of the same video with different resolutions) I did not have that problem. I mean the 720p version seems to not have issues no matter what the resolution of my monitor is, whether it be my native resolution or a smaller one. I have also tested this with 1080p videos and I again do not encounter that issue. Anything bigger than 1080p and green everywhere.