Extreme Lag & CPU Usage (1080p = PC Killer)

[Pecacheu]

I'm using this library make an application to view and record from multiple cameras at the same time, sort of like a virtual camera DVR system.

I started with Adobe AIR, but Flash has a terrible camera API, and there's no way to detect camera plugging in/unplugging, so I ported all the code to Java. It does work, but unfortunately it lags WAY too much to be useful. I'm getting 6 to 10 redraws/sec at 720p with just the 2 built-in ones, and with 6 cameras at 1080p, which is what I want to support, the lag is just insane. The fans kick in, the CPU goes into overdrive, and if you leave it running long enough, you'll have yourself a great glass cook-top.

Hmm... simple camcorder app using more resources that CoD Black Ops 3? Yeah that sounds about right.

Seriously though, even when using the native driver, and with just one camera at 1080p, it still lags badly and takes CPU usage to 100% consistently on all 4 cores. Is there just no hardware acceleration or something?

It's a great library, if only I could actually use it without setting the resolution to less than 640x480. And this isn't a Raspberry Pi or anything, this is a high-end gaming PC. (well, technically gaming tablet PC)

The AIR version can handle 6 webcams fine without lag on the same device. (Although, sometimes the internal cameras don't want to work with external ones connected. Again, the Flash camera API is terrible.)

EDIT: I just found out that with native camera driver, using literally any resolution other than 1080p (including higher at 1920x1440), significantly reduces the lag. It also gets rid of the random connection issues that occur when set to 1080p. I guess it's just an HDTV hater? The default driver still lags severely even at lower resolutions, though.

[sarxos]

Hi @Pecacheu,

In regards to your question:

Simple camcorder app using more resources that CoD Black Ops 3? Yeah that sounds about right.

If you consider what components and processes are used in these products?

Does CoD Black Ops 3 transfers megabytes of data from USB and copy memory from native to direct buffers and to offscreen using processor? Nope. It uses GPU for most of the load.

Does USB uses hardware acceleration or GPU for data transfer? This would be nice, but unfortunately USB 2.0 uses CPU-based polling model in order to receive device communications. You can imagine CPU asking USB controller "Is it done?" over and over again. This wastes a load of CPU juice.

When you add things like memory operations, e.g. copy from webcam buffers, to direct buffers, then to heap, then again to heap (e.g. image resize which is the most expensive operations of all), and finally to offscreen. This is all done by CPU. Only drawing can be sometimes accelerated. All other operations are done by CPU. I'm sorry to disappoint you but Java was never good at this.

Is there just no hardware acceleration or something?

Java has "some" hardware acceleration, but it works in some cases, not always, so forget about features from Black Ops 3 where GPU is used for 95% of work. Java uses hardware acceleration only for drawing, and only in some case when buffer is not modified (e.g. resized). If you touch even one pixel from the buffer Java will silently give up trying to accelerate it.

And this isn't a Raspberry Pi or anything, this is a high-end gaming PC.

You may have the best graphic card in the world, 1024 GB of RAM, but bottleneck will always be USB controller and CPU.

sometimes the internal cameras don't want to work with external ones connected

This is probably due to USB capacity limit (60 MB/s).

I just found out that with native camera driver, using literally any resolution other than 1080p (including higher at 1920x1440), significantly reduces the lag. It also gets rid of the random connection issues that occur when set to 1080p. I guess it's just an HDTV hater?

That's really hard to tell. I'm not sure what is the difference, but maybe for 1080p different format is used than for the other ones, e.g. RGB instead of JPG.

The default driver still lags severely even at lower resolutions, though.

Yes, default driver was never the best one to use, but it was the most portable one. You can use it on almost every OS and that's why it'd default. The webcam-capture-driver-native performs much better since it uses JNI instead of BridJ and use modern framework under the hood, namely libyuv instead of outdated libvideoinput which is used by OpenIMAJ in default driver on Windows.

[sarxos]

And in regards to:

I just found out that with native camera driver, using literally any resolution other than 1080p (including higher at 1920x1440), significantly reduces the lag.

Maybe it's worth to ask directly https://github.com/frankpapenmeier/webcam-capture-driver-native

[Pecacheu]

I will concede your point about video games using the GPU mostly rather than than CPU.

Also, the issue with Adobe AIR was definitely not a USB bandwidth issue, since it only happens in the latest version.

Anyway, the native driver mostly works well, so I'll probably just use that. It does error out (a NullPointerException, I think) every once and a while while opening a Webcam, which has happened with only one camera connected, but otherwise it usually works.

I also just tested on my (not as powerful as the tablet) MacBook Pro, and things don't go quite so well there.

On the default driver, even with only the built-in iSight at low-res, it's already full-blast with the fans, and it's getting hot FAST.

As for the native driver, there's no lag, including at 1080p, but it has a lot more connection issues on the Mac, and it doesn't even recognize the built-in iSight. It also crashes the whole app if no cameras are connected, and occasionally spews out an exception if one is unplugged. I could probably just open a new issue on the native driver Github for that.

Oh, also is there a way to draw over the camera view? Currently, I'm hijacking the paintComponent method by extending the WebcamPanel class, but it draws my geometry offset by the X,Y of the component, and I can't figure out how to correct that.

[sarxos]

In regards to:

It does error out (NullPointerExcepton, I think) every once and a while while opening a Webcam

I would create issue with detailed explanation, code sample, stack trace and debug info. You can make this project better just by doing this because good software cannot be build without a good testers.

Resolving issues is another matter. Often problems reported by one user cannot be reproduced by the other one. Problems differs from system to system and strongly depend on the hardware. This is where people with programming skills are welcome. These projects are open source, to be used for free by everyone and from everywhere. Therefore everyone with programming skill can improve them and fix bugs.

On the default driver, even with only the built-in iSight at low-res, it's already full-blast with the fans, and it's getting hot FAST.

Since I do not own Mac I'm pretty sure there can be some bugs and performance issues. When I was writing webcam-capture code I was able to test only on Windows and Linux. The only fixes I did for Mac was when I borrowed it from my colleague for several days but I haven't observe CPU overload in any scenario in that time. However, I'm aware that other users reported problems on newer Macs (e.g. https://github.com/sarxos/webcam-capture/pull/257) but I have no way to verify and test it.

Oh, also is there a way to draw over the camera view? Currently, I'm hijacking the paintComponent method by extending the WebcamPanel class, but it draws my geometry offset by the X,Y of the component, and I can't figure out how to correct that.

Yes there is. You have two options:

1. Use WebcamImageTransformer to draw on the image before it's returned from webcam, so when you invoke webcam.getImage() the changes in image are already made, or
2. Use WebcamPanel.Painter interface to draw on WebcamPanel's Graphics2D without a need to override paintComponent(..) or extending WebcamPanel itself.

Examples:

1. Webcam Capture Image Transformer Example (check this file for more details, especially take special care to check how transform(BufferedImage) method work and webcam.setTransformer(...))
2. Webcam Capture Custom Painter Example (check MyPainter class in this file and two methods: paintPanel(..) and paintImage(..))

These two approaches differs in such a way that the first one (namely image transformation) draws directly in image and modifies it, and the second one (namely panel painter) draws on a component (panel itself) and do not modify image.

[Crigges]

I didn't read everything, (just the first post) but i was trying to build something similar, with the same issue. Iam NO expert to this libary but i try to explain what I think is going wrong: This library is using Java Swing for the Panels which is using some kind of Hardware acceleration in the background. However this acceleration is horrible slow and causes a huge CPU overhead. Addionaly you can't realy access this accelleration asynchronously. So you have one single thread which needs to resize and draw all of your images with realy poor GPU acceleration. I was gaining signifficant fps gains by using a CPU based Multithreaded resizing method, but painting was still too slow.

So I moved away from Swing and the JMF to get better GPU accelleration using OpenGL, with great success. I can record and display 4x 1080p cams @60fps at the same time, without any lags. You can clone this repo: https://github.com/Crigges/ShootAlyzzerGL if you want to see the full project or just take a look @ https://github.com/Crigges/ShootAlyzzerGL/blob/master/core/src/systems/crigges/gui/CameraActor.java and https://github.com/Crigges/ShootAlyzzerGL/blob/master/core/src/systems/crigges/gui/CameraController.java to see how i load the RecordedPixels into the GPU memory.

[Pecacheu]

Thanks for the suggestion. In fact, I spent the last 3 days (except for yesterday, I had to help set up the Christmas lights) porting the code to use LibGDX. Naturally, most of that time was spent googling for help and errors. From your code I used most of your draw() method from CameraActor.java and a bit of the ImageFetcher class from CameraController.java. I couldn't get any menu drop-down interfaces of LibGDX to work no matter what I tried, so for the menu bar I just used my existing Swing code, and made the JFrame fullscreen, always-on-top, and transparent, so it looks like a Mac menu bar at the top of the screen.

Anyway, LibGDX does not disappoint, significantly improving the performance of the default driver, getting nearly 30 fps with 2 cameras 1080p, and even 6 at 1080p only takes it down to 12 or 15 fps. That's even better than the native driver was before! Not as good as the buttery-smooth Adobe AIR version, but still, impressive.

I can't test it with the native driver, though. If I turn on the native driver, I get this error on a line where I call synchronized(camSync) { cam.getImageBytes(buf); }. The thread in question doesn't do anything but call that every so often, then calculate average the FPS over the last 5 frames. And the buffer should be the right size, assuming that cam.getViewSize() (called after webcam was opened) reported the correct dimensions.

[CamView3] Video thread for 'Rear Camera' error:
com.github.sarxos.webcam.WebcamException: Cannot execute task
        at com.github.sarxos.webcam.WebcamProcessor$AtomicProcessor.process(WebcamProcessor.java:72)
        at com.github.sarxos.webcam.WebcamProcessor.process(WebcamProcessor.java:140)
        at com.github.sarxos.webcam.WebcamTask.process(WebcamTask.java:46)
        at com.github.sarxos.webcam.ds.cgt.WebcamReadBufferTask.readBuffer(WebcamReadBufferTask.java:22)
        at com.github.sarxos.webcam.Webcam.getImageBytes(Webcam.java:722)
        at com.pecacheu.camview.VideoPane.lambda$0(Camera.java:109)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.BufferOverflowException
        at java.nio.DirectByteBuffer.put(Unknown Source)
        at com.github.sarxos.webcam.ds.nativeapi.NativeWebcamDevice.getImageBytes(NativeWebcamDevice.java:168)
        at com.github.sarxos.webcam.ds.cgt.WebcamReadBufferTask.handle(WebcamReadBufferTask.java:41)
        at com.github.sarxos.webcam.WebcamProcessor$AtomicProcessor.run(WebcamProcessor.java:81)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 more

[sarxos]

Hi @Pecacheu,

Just to clarify. You wrote:

LibGDX does not disappoint, significantly improving the performance of the default driver, getting nearly 30 fps with 2 cameras 1080p, and even 6 at 1080p only takes it down to 12 or 15 fps. That's even better than the native driver was before!

Please note that driver performance is not related to drawing at all and therefore LibGDX does not increase driver performance. The driver job is to download image from webcam and only that. Different drivers differs in such a way that the download speed (via UVC interface) is faster or slower. It's also measured in FPS, but this is how fast frames can be downloaded from webcam, not how fast frames can be painted. Just to summarize - it's your drawing code performance increased by LibGDX, not the driver performance.