Bayer (RAW) Mode

[rovarma]

I've rewritten most of the camera initialization, register setup, etc to put the camera in a mode where it outputs Bayer RAW (8bpp) instead of YUV (16bpp).

This significantly decreases (halves) required bandwidth and thereby increases performance. Especially setups where multiple cameras are used on 1 machine perform significantly better.

I've also added support for more FPS modes, including 640x480@75 FPS.

The API has changed slightly:

• getFrame no longer returns a pointer to a newly-allocated block of memory. Instead, the user is responsible for providing getFrame with an appropriately-sized buffer. This increases performance of the driver by cutting out intermediate buffers and memcpy's.
• It's now possible to specify the output mode that you want through the init call. Depending on the output mode selected, the buffer provided to getFrame must be sized correctly. Supported are:
  • Bayer RAW: The buffer provided to getFrame must be width * height bytes large
  • BGR Color: The buffer provided to getFrame must be width * height * 3 bytes large
  • RGB Color: The buffer provided to getFrame must be width * height * 3 bytes large

To support the color output, I've added a function to convert from Bayer to BGR/RGB. It's a bit slow (~0.35ms for a single frame @ 640x480), but it's probably good enough for most use cases.

If the performance needs further improvement, I might convert that to SSE at some point. Or I might not...we'll see.

As a sidenote, this seems to have fixed the issue that @HipsterSloth tried to fix in https://github.com/inspirit/PS3EYEDriver/pull/29 so if he agrees, I think we can close that PR.

Thanks to @cboulay and @HipsterSloth for beta testing.

[HipsterSloth]

Yeah I haven't seen the crash that PR #29 was supposed to fix since using the new Bayer filter. I suspect the crash probably is still lurking in libusb and we are just missing it now that the camera bandwidth has dropped substantially.

Further, after trying to use #29 in PSMoveService for a while I noticed occasional bad perf camera issues that I suspect are related to the lock around the control transfer request queue. So PR #29 isn't a good fix anyway and for that reason alone I think we should kill off my PR.

[cboulay]

Congratulations @rovarma . This is truly excellent.

I just pulled down your fork/branch and tested the sdl example on my macbookpro. It definitely works better for me than the current version.

I noticed a couple small things, none of which would stop me from using this:

• I was getting occasional screen tearing but this is to be expected because vsync is turned off.
• At 640x480@75fps, I get a slightly discolored (darker) horizontal band just below the middle of the image. I don't see this at 60 fps.
• Also @75, but not @60, there are some vertical discoloration stripes of blue and green.
• You took out the OpenCV example! That might be useful to see if the artifacts are the same when using OpenCV's different Bayer2BGR modes.
• I guess the exposure is constant-ish, so lower fps has longer frame times and thus appears much brighter. But I don't think it should be up to PS3EYEDriver to reset the exposure depending on fps.

I wonder if the artifacts I'm seeing at @75 fps are always there but not visible at lower frame rates because of the higher exposure / longer frame times.

Edit: Let me know if you don't also see the very thin dark band then I'll take a screenshot.

[rovarma]

• At 640x480@75fps, I get a slightly discolored (darker) horizontal band just below the middle of the image. I don't see this at 60 fps.
• Also @75, but not @60, there are some vertical discoloration stripes of blue and green.

Yep, I also get these artifacts. I think (but am not sure) that these are caused by the camera itself. I believe last time I tried it, I had the same problem with OpenCV, so I don't think it's my Bayer -> BGR code. But I might be wrong.

As a verification step: if you happen to have the CL Eye Driver running, maybe you could try running one of their test apps at 75 FPS to see if the same artifacts appear? Not sure how much work that is for you.

• You took out the OpenCV example! That might be useful to see if the artifacts are the same when using OpenCV's different Bayer2BGR modes.

Yeah, I removed it because there seemed little point in it; it does not function at higher framerates as we saw, because opencv is too slow. It seemed to make more sense to move the functionality I had in there to the SDL app.

You may be right though that it could be useful to serve as a reference color conversion test thing. Maybe I'll add it back in a stripped down form.

• I guess the exposure is constant-ish, so lower fps has longer frame times and thus appears much brighter. But I don't think it should be up to PS3EYEDriver to reset the exposure depending on fps.

Good point about the exposure. I also noticed the image got progressively darker as you go up in frame rate. Your explanation makes sense. I agree though that we shouldn't try to fix this in the driver.

[rovarma]

Just did a quick test with the opencv app, using opencv's Bayer -> BGR code. The artifacts are also present there. That doesn't mean I haven't done something else wrong, but at least the color conversion code seems in order.

[rovarma]

@cboulay: tested the CLEyeDriver @ 75 FPS. It does not have the darker horizontal line about halfway through the image.

It does have the vertical blue/green stripes; I think those are related to exposure somehow.

Will need to investigate further, but is a minor issue as far as I'm concerned.