Queue macOS input reports so that large responses aren't dropped

[GregDomzalski]

Description

This has been a longstanding bug in the SDK that has been haunting me since the FIDO device code was originally written. Apple's documentation on how async input reports are intended to work are about as clear as mud. But with some extra (temporary) debug logging I was able to finally reproduce the issue with enough visibility into what was actually happening. The implementation used by our own libfido2 library now makes a lot more sense as well.

Fundamentally, we were just using the IOHIDDeviceRegisterInputReportCallback and its corresponding callback incorrectly. The read buffer that we supply during the registration is just meant to be a place for macOS to cache the report data... I guess? The intended use still is not super clear to me.

But what was clear from the logs is that the report callback was getting called faster than we were calling GetReport to read said reports. And since we previously did not queue up any of the reports until they were read, it resulted in reports either getting dropped, or arriving before we ran the IO runloop. That would usually result in the CFRunLoopRunInMode call inside of GetReport to timeout. Not because the YubiKey was taking a long time, but because the report already came and went and we missed it.

So now, we pass a ConcurrentQueue into the native callback via a GCHandle. We queue the report that we see in the callback so that when GetReport is eventually called, we will have the report. We first attempt to read straight from the queue, and if there's nothing there, we run the runloop which should then cause macOS to read from the YubiKey. This behavior now much more closely matches libfido2.

Fixes (internal issue)

Type of change

Please delete options that are not relevant.

• [x] Bug fix (non-breaking change which fixes an issue)

How has this been tested?

Ran some local FIDO2 tests as well as real-world use cases using our internally dependent software.

Test configuration:

• Firmware version: 5.4.3
• Yubikey model: YK5 ANFC

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [ ] I have run dotnet format to format my code
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes

[github-actions[bot]]

Test Results: Windows

    2 files      2 suites   2s :stopwatch: 3 615 tests 3 615 :white_check_mark: 0 :zzz: 0 :x: 3 617 runs  3 617 :white_check_mark: 0 :zzz: 0 :x:

Results for commit 5c10887f.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

Test Results: Ubuntu

    2 files      2 suites   5s :stopwatch: 3 607 tests 3 607 :white_check_mark: 0 :zzz: 0 :x: 3 609 runs  3 609 :white_check_mark: 0 :zzz: 0 :x:

Results for commit 5c10887f.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

Test Results: MacOS

    2 files      2 suites   1s :stopwatch: 3 607 tests 3 607 :white_check_mark: 0 :zzz: 0 :x: 3 609 runs  3 609 :white_check_mark: 0 :zzz: 0 :x:

Results for commit 5c10887f.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

Package	Line Rate	Branch Rate	Complexity	Health
Yubico.Core	42%	31%	4257	➖
Yubico.YubiKey	51%	47%	18417	➖
Summary	49% (31675 / 64317)	45% (8017 / 17979)	22674	➖

Minimum allowed line rate is 40%