HoloLens 2 and Task Support

[jbienzms]

This is a pretty big PR so I want to fully document the reason for each of the changes.

Main Work

I started this work because I could not get MRLT to work on HoloLens 2 without errors in the Unity log and on screen. The most prominent errors were:

HoloLens locatable camera has failed to set auto exposure off
HoloLens locatable camera has failed to set exposure to ## as requested
HoloLens locatable camera has failed to set auto WhiteBalance off
HoloLens locatable camera has failed to set WhiteBalance to #### as requested
[CameraCapture] Can't get camera matrix!!

It turned out that these errors are primarily due to HoloLens 2 wanting to use ExposureControl rather than Exposure and wanting to use WhiteBalanceControl rather than WhiteBalance.

My first attempts at fixing this looked promising, but seemed to only work randomly. Now I started seeing new errors in the log:

HoloLens locatable camera has failed to set exposure to 00:00:00.0101000. System.NullReferenceException: Object reference not set to an instance of an object.
HoloLens locatable camera has failed to set ISO to 80. System.ArgumentException: No capture devices are available.

These ended up stemming from two different core issues:

1. In HoloLens 2, the Control interfaces can only be adjusted while the camera is streaming.
2. It turned out there were several possible race conditions when setting these values (due to callbacks) and when shutting down or starting LightCapture back up.

Only adjusting the control values while streaming was a fairly easy fix. Apply them only if the camera is running, otherwise cache the values for the next time the camera is started.

The race conditions and callbacks ended up being responsible for the bulk of the refactoring. The root of the problem is that methods like Initialize, SetExposure and SetWhiteBalance all returned void, but internally they ran a bunch of callbacks. This made those methods function like async void. Now, I'm not going to say that async void should never be used, but it tends to be the source of many headaches, race conditions and threading bugs. This is because the caller has no way to know when the function has actually completed. For more information see Avoid Async Void.

It's also worth pointing out that any method that returns a Task can be treated as void, just not the other way around.

The following line:

var t = SetWhiteBalanceAsync(5000);

Is functionally equivalent to:

SetWhiteBalance(5000, myCallback);

If Task t is never awaited, it will run in the background and we'll never know whether it failed or completed. This would be equivalent to myCallback never getting called but the application being unaware it didn't happen.

So, every method in the library that previously used a callback was converted to return a Task. And every place that called one of those methods was also converted to a Task (whenever possible). There were a few cases where this wasn't possible. For example, when responding to a UI button click. In these "last mile" cases async void was used, but error handlers were employed.

Other Fixes and Changes

Those are the main things that were refactored. In addition there were also other fixes and improvements which I'll cover now:

• Originally, code to change Exposure, White Balance and ISO was conditionally compiled to only run with UWP. I created a new interface ICameraControl that any camera controller on any platform can implement. Currently only UWP does, but the example now checks for this interface and will use it if available.

• In the NativeArray<Color24> method of CameraCaptureWebcam there was a blind cast of resizedTexture to Texture2D. However, at least on my system resizedTexture is a WebCamTexture which does not inherit from Texture2D. This would have resulted in an InvalidCastException. For now I changed this to an As instead of a hard cast and throw an UnsupportedException with a clear message if the conversion can't be done.

• In LightCapture.Update, the Map was being used before the camera had fully completed initializing. The reason this wasn't an error before was probably again due to callbacks. The camera was probably partially initialized (up to the first callback) but not completely initialized. This became apparent when initialization became a Task and that Task did not complete until the device was fully initialized. The fix was a simple check to make sure Map is not null before reading it.

• CameraCaptureWebcam start delay was updated from 0.5 seconds to 2.0. I have a Logitech BRIO and it takes much longer than 0.5 seconds to initialize.

• The range values for Exposure of -10 to 10 did not work for all devices and did not map easily to the new Control interfaces. I changed Exposure to a floating point value ranging from 0.0 to 1.0. This allows me to directly map the values to the Min and Max values returned by the Controller. It can also be easily converted back to the -10 to 10 range for the old Exposure interface.

• ISO of 80 was way too dark. From reading online, it looks like this was probably supposed to be 800 and not 80 so I changed it to 800. This was probably never caught because I don't think ISO was actually ever successfully set before. ISO Control can only be adjusted while streaming.

• Upped the exposure from 0.17 to 0.2 (equivalent of changing from -7 to -6). This was due to getting ISO working. An ISO of 800 is recommended for indoors, but the exposure at this ISO was a little too dark. This new default looks good on HoloLens 2.

• Added Min and Max values for Exposure, ISO and White Balance to the Example scene.

• Converted voice commands in the Example scene to constants so they couldn't get out of sync.

• Added voice commands to the Example scene to allow for adjusting ISO.

[maluoi]

This is awesome work, and fantastic documentation! Thanks so much for doing this :)

I'll try and put in some time to check corners like AR Foundation, and put this all into a release.