StackOverflow exception on MacOS

[xfischer]

Hi ! Thanks for this useful project, however I cannot get the samples to run.

I have an StackOverflow exception here and the runtime crashes :

// File Device.cs

// Find controls.
_controls = inputParsers.Values
                .SelectMany(inputParser => Enumerable.Range(0, inputParser.ValueCount)
                    .Select(index => (index, dataValue: inputParser.GetValue(index))))
                .ToDictionary(t => (t.dataValue.DataItem, t.index), t => new Control(this, t.dataValue, t.index));
`

[thargy]

Hi @xfischer,

It's great to see someone using it. Thanks for reporting the error. I use it regularly on another project and although I've tested with quite a few devices, I can't test with everything. There's currently still a dependency on HIDSharp which throws a lot of NotSupportedExceptions and is particularly well maintained. Eventually, I might get round to replacing the dependency entirely.

As I can't reproduce this, it's almost certainly a particular device on your machine that is causing the problem.

Are you sure it was a StackOverflow exception as there's nothing obviously recursive here? Do you have a StackTrace? If not, run in debug and set to stop on StackOverflow exceptions, you should be able to get the last few frames of the stack there. That will help narrow down the loop that's causing the overflow.

It would also be useful to unplug any peripherals you have and see if that helps, by adding back one at a time you will be able to identify the peripheral that is causing the crash. That might give us a hint too.

[xfischer]

Hi @thargy, I can't set an exception catch point because the runtime crashes after executing the line I have copied on the 4th device.

I have tried to unplugged everything connected to my MacBook (even the power cable), and I have the same issue. However (that's interesting) : running the same sample from a Parallels VM running Win10 on the same MacBook runs perfectly.

[thargy]

Hi @xfischer,

Sorry for the delay, I can't test on a Mac as I don't have access to one, so I've performed a detailed manual review. However, all the code in HIDDevices is cross-platform, only HIDSharp has platform-specific code in it. That strongly suggests the problem is there.

Sadly, there doesn't appear to be a viable candidate for a StackOverflowException. All the access to HIDSharp code goes through simple properties on data that is initialised on construction, and those constructors should already have run before this line. That (unusually) doesn't give me an obvious candidate to investigate.

The next step would be to try and figure out where the crash is occurring on your machine. Step one is to create a minimal reproducible example.

Create a blank .NET Console project of the same framework as you're currently experiencing a problem in and include the HIDSharp NuGet version 2.1.0:

<PackageReference Include="HidSharp" Version="2.1.0" />

Then add the following code to the Main() method (this is effectively the main loop cut down to get us to the problem line):

foreach (var hidDevice in DeviceList.Local.GetHidDevices())
{
    Console.Write($"Parsing {hidDevice} ... ");
    try
    {
        var rawReportDescriptor = hidDevice.GetRawReportDescriptor();
        var reportDescriptor = new ReportDescriptor(rawReportDescriptor);
        var deviceItems = reportDescriptor.DeviceItems;
        var inputParsers = deviceItems.ToDictionary(i => i, i => i.CreateDeviceItemInputParser());
        var test = inputParsers.Values
                    .SelectMany(inputParser => Enumerable.Range(0, inputParser.ValueCount)
                        .Select(index => (index, dataValue: inputParser.GetValue(index).DataItem)))
                    .ToArray();
        Console.WriteLine($"Success: {test.Length} controls found.");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error: {ex.Message}");
    }
}

On my machine this produces the following output (those errors are expected and due to the limits of HIDSharp):

Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Success: 0 controls found.
Parsing Logitech Logitech Extreme 3D (no serial number) (VID 1133, PID 49685, version 2.4) ... Success: 17 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Error: Unable to reconstruct the report descriptor.
Parsing Logitech Audio Controls (no serial number) (VID 1133, PID 2591, version 1.0) ... Success: 13 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Error: Unable to reconstruct the report descriptor.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Error: Unable to reconstruct the report descriptor.
Parsing (unnamed manufacturer) (unnamed product) (no serial number) (VID 1699, PID 20513, version 0.1) ... Success: 8 controls found.
Parsing Logitech Audio Controls (no serial number) (VID 1133, PID 2591, version 1.0) ... Success: 1 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Success: 0 controls found.
Parsing Microsoft Natural® Ergonomic Keyboard 4000 (no serial number) (VID 1118, PID 219, version 1.73) ... Success: 8 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Success: 8 controls found.
Parsing (unnamed manufacturer) Controller (Xbox One For Windows) (no serial number) (VID 1118, PID 767, version 0.0) ... Success: 22 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Success: 20 controls found.
Parsing Microsoft Natural® Ergonomic Keyboard 4000 (no serial number) (VID 1118, PID 219, version 1.73) ... Success: 4 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 12.9) ... Success: 0 controls found.
Parsing Saitek Saitek X52 Pro Flight Control System (no serial number) (VID 1699, PID 1890, version 0.30) ... Success: 49 controls found.
Parsing (unnamed manufacturer) (unnamed product) (no serial number) (VID 1699, PID 20609, version 0.1) ... Success: 0 controls found.
Parsing (unnamed manufacturer) (unnamed product) (no serial number) (VID 1699, PID 20545, version 0.1) ... Success: 8 controls found.

Hopefully, that will throw the StackOverflowException on your Mac (but not the Win VM). At that point, we've identified the bug is in HIDSharp. If it doesn't error, report back as we've got a much more subtle problem that's going to take a bit more work to track down.

Once you've identified the device that causes the crash (it will be the last incomplete Parsing ... line before the crash), you can test just that device by taking its VID number, e.g.:

DeviceList.Local.TryGetHidDevice(out var hidDevice, 1699);
Console.Write($"Parsing {hidDevice} ... ");
try
{
    var rawReportDescriptor = hidDevice.GetRawReportDescriptor();
    var reportDescriptor = new ReportDescriptor(rawReportDescriptor);
    var deviceItems = reportDescriptor.DeviceItems;
    var inputParsers = deviceItems.ToDictionary(i => i, i => i.CreateDeviceItemInputParser());
    var test = inputParsers.Values
                .SelectMany(inputParser => Enumerable.Range(0, inputParser.ValueCount)
                    .Select(index => (index, dataValue: inputParser.GetValue(index).DataItem)))
                .ToArray();
    Console.WriteLine($"Success: {test.Length} controls found.");
}
catch (Exception ex)
{
    Console.WriteLine($"Error: {ex.Message}");
}

Next, it would be worth downloading a copy of HIDSharp 2.1 from this link and including the source directly in your project (rather than as a NuGet). That will then allow you to step through the code easily and detect where recursion is occurring. If you can identify the cause we can report it to the original author (I'm not sure how active he is) and I can see if there is a way to work around the issue. Please let me know how you get on!

[xfischer]

Hi, thanks for the debbuging tips. Here's what I have (no exception :( ) I'll try to run your project with HIDSharp source instead of the nugget and let you know

Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found.
Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found.
Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found.
Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, version 1.1) ... Success: 3 controls found.
Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 5 controls found.
Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 271 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 24.7) ... Success: 0 controls found.
Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, version 1.1) ... Success: 52 controls found.
Parsing Apple, Inc Apple Keyboard (no serial number) (VID 1452, PID 545, version 0.69) ... Success: 7 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 24.7) ... Success: 930 controls found.
Parsing Apple, Inc Apple Keyboard (no serial number) (VID 1452, PID 545, version 0.69) ... Success: 265 controls found.
Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 24.7) ... Success: 173 controls found.

[thargy]

Well, that's unexpected! My next best guess is that the static initialisation of Usages is at fault due to runtime differences in the initialisation of statics on Mac. My NuGet does push source with it so you should be able to enable source stepping in Visual Studio if you're using that. Otherwise, cloning the source and linking directly to it will work.

If you can find the stack frames by stepping through that will be extremely useful.

[thargy]

Also please be sure you are on the latest version of the NuGet, as there was an issue with Usages initialisation that was fixed in an earlier version.

[xfischer]

@thargy I have forked the repo, referenced directly HidSharp from the source you have provided. Still the exception, but I have narrowed down the path where it occurs.

https://github.com/DevDecoder/HIDDevices/blob/31d0e608e02c1566a68d5954e9bf654c94f14db4/HIDDevices/Control.cs#L28

[thargy]

Yes, that is a problem with static initialisation. It shouldn't be a problem but the order of static initialisation in C# is unspecified and must be different in the MAC runtime. There was a related issue that was fixed in ca136b1, so double check you are on the latest version (tbf you probably are if you've done a clean checkout!).

I'll have a look to see if I can suggest a code change to try. Thanks for helping with this issue.

[thargy]

@xfischer, please can you try executing just the following code from the Main() method? (Note do not do any other code prior to this):

Console.WriteLine(Usage.Get(0x10001u));

The output should be

GenericDesktop - Pointer

If that crashes, we have confirmed the diagnosis.

[thargy]

If for some reason that doesn't crash, try:

foreach (var hidDevice in DeviceList.Local.GetHidDevices())
{
    Console.Write($"Parsing {hidDevice} ... ");
    try
    {
        var rawReportDescriptor = hidDevice.GetRawReportDescriptor();
        var reportDescriptor = new ReportDescriptor(rawReportDescriptor);
        var deviceItems = reportDescriptor.DeviceItems;
        var inputParsers = deviceItems.ToDictionary(i => i, i => i.CreateDeviceItemInputParser());
        var test = inputParsers.Values
                    .SelectMany(inputParser => Enumerable.Range(0, inputParser.ValueCount)
                        .Select(index => (index, dataValue: inputParser.GetValue(index).Usages.ToArray())))
                    .ToArray();
        Console.WriteLine($"Success: {test.Length} controls found.");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error: {ex.Message}");
    }
}

Note, the only change here is to grab the Usages as an array in the final select, which is equivalent to the line you've indicated. If it crashes here, we're back to HIDSharp, if it doesn't, the following should crash:

...
                        .Select(index => (index, dataValue: inputParser.GetValue(index).Usages.Select(Usage.Get).ToArray())))
...

As this now does exactly the same thing as the line you've indicated.

[xfischer]

Console.WriteLine(Usage.Get(0x10001u));

• No crash. Output 65537

• Following test (the big foreach) : OK, no crash.

• Replacement with .Select(index => (index, dataValue: inputParser.GetValue(index).Usages.Select(Usage.Get).ToArray()))) -> Crash

Hope that helps !

[thargy]

Thanks @xfischer,

Obviously, I assume that code doesn't crash on the Windows VM? Frankly, the issue is a bit weird.

Here's my theory, it doesn't look like calling Usage.Get is intrinsically crashing, so maybe there is a specific Usage, that one of your peripherals has, that is causing a recursion to occur somehow? It would have been good to see the output prior to the crash, does it always crash at the same point (i.e. on the same device).

Let's add a bit more debug logic. Expand Select(Usage.Get) into a lambda to output the usage number:

.Select(u =>
{
    OutputHelper.WriteLine($"0x{u:x}u ");
    return Usage.Get(u);
})

Please paste the output here, in case I can see any clues.

If that always crashes after the same usage number, then we're onto something. Or if always crashes on the same usage page (the top 2 bytes, e.g. ignore the last 4 hex digits 0x1xxxx, or 0xff10xxxx). Say it crashes at 0x10039u, then try:

Console.WriteLine(Usage.Get(0x10039u);

On its own, if that crashes we have our culprit, and I can look at what's different about that code. It would be handy if you try and get something else from that UsagePage too to try and see if it's the page or the Usage. You can search for the page/usage in GeneratedUsagePages.generated.cs (which is a large file so open in something like Visual Studio Code).

If that doesn't crash at the same point, then try adding:

Console.WriteLine(Usage.Get(0x10001u);

Before the for loop, as that will force static initialisation prior the for loop.

It's a strange one, but we seem to be circling the drain now. I hope it doesn't prove to be one of those "I'm an idiot" moments (highly likely in my experience!)

[xfischer]

Yes, it's the same value everytime. here's the output

Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, version 1.1) ... Success: 3 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 271 controls found. Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 24.7Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, ver sion 1.1) ... Success: 3 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 271 controls found. Parsing Logitech USB Receiver (no serial number) (VID 1133, PID 50475, version 24.7 Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number ) (VID 1452, PID 631, version 8.96) ... Success: 0 controls found. Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, ver sion 1.1) ... 0x2004d1u 0xff120031u 0xff120051u Success: 3 controls found. Parsing Apple Inc. Apple T1 Controller (no serial number) (VID 1452, PID 34304, ver sion 1.1) ... 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xd0038u 0xd0033u 0xd0032u 0x10030u 0x10031u 0xff110001u 0xff110002u Success: 52 controls found. Parsing (unnamed manufacturer) Apple Internal Keyboard / Trackpad (no serial number) (VID 1452, PID 631, version 8.96) ... 0x700e0u 0x700e1u 0x700e2u 0x700e3u 0x700e4u 0x700e5u 0x700e6u 0x700e7u 0x70000u 0x70001u 0x70002u 0x70003u 0x70004u 0x70005u 0x70006u 0x70007u 0x70008u 0x70009u 0x7000au 0x7000bu 0x7000cu 0x7000du 0x7000eu 0x7000fu 0x70010u 0x70011u 0x70012u 0x70013u 0x70014u 0x70015u 0x70016u 0x70017u 0x70018u 0x70019u 0x7001au 0x7001bu 0x7001cu 0x7001du 0x7001eu 0x7001fu 0x70020u 0x70021u 0x70022u 0x70023u 0x70024u 0x70025u 0x70026u 0x70027u 0x70028u 0x70029u 0x7002au 0x7002bu 0x7002cu 0x7002du 0x7002eu 0x7002fu 0x70030u 0x70031u 0x70032u 0x70033u 0x70034u 0x70035u 0x70036u 0x70037u 0x70038u 0x70039u 0x7003au 0x7003bu 0x7003cu 0x7003du 0x7003eu 0x7003fu 0x70040u 0x70041u 0x70042u 0x70043u 0x70044u 0x70045u 0x70046u 0x70047u 0x70048u 0x70049u 0x7004au 0x7004bu 0x7004cu 0x7004du 0x7004eu 0x7004fu 0x70050u 0x70051u 0x70052u 0x70053u 0x70054u 0x70055u 0x70056u 0x70057u 0x70058u 0x70059u 0x7005au 0x7005bu 0x7005cu 0x7005du 0x7005eu 0x7005fu 0x70060u 0x70061u 0x70062u 0x70063u 0x70064u 0x70065u 0x70066u 0x70067u 0x70068u 0x70069u 0x7006au 0x7006bu 0x7006cu 0x7006du 0x7006eu 0x7006fu 0x70070u 0x70071u 0x70072u 0x70073u 0x70074u 0x70075u 0x70076u 0x70077u 0x70078u 0x70079u 0x7007au 0x7007bu 0x7007cu 0x7007du 0x7007eu 0x7007fu 0x70080u 0x70081u 0x70082u 0x70083u 0x70084u 0x70085u 0x70086u 0x70087u 0x70088u 0x70089u 0x7008au 0x7008bu 0x7008cu 0x7008du 0x7008eu 0x7008fu 0x70090u 0x70091u 0x70092u 0x70093u 0x70094u 0x70095u 0x70096u 0x70097u 0x70098u 0x70099u 0x7009au 0x7009bu 0x7009cu 0x7009du 0x7009eu 0x7009fu 0x700a0u 0x700a1u 0x700a2u 0x700a3u 0x700a4u 0x700a5u Stack overflow.

Annnnnnnnddd it's crashing now on a single call to Console.WriteLine(Usage.Get(0x700a5u));

I think we have it :) Now what can I do to test more ? Maybe we can set a TeamViewer so that I can let you test on my mac ?

[thargy]

Ironically, I found the bug just before you posted! I'll push a fix soon.

[xfischer]

What a journey !!!!

[thargy]

Thanks @xfischer, it was indeed an "I'm an idiot" moment.

0xa5 doesn't currently appear in the specs for Keyboard-Keypad (0x7) (at least not in the usage tables I use). When you ask for an unknown usage it dynamically creates one for you, however, this logic was broken in the recent refactor and the lack of tests didn't pick it up! If we can figure out what that usage actually is for, I'll let @IntergatedCircuits know as they seem amenable to accepting pull requests, for now, it will report as Unknown (though it will, of course, parse the page still).

The working on Win VM was a red herring that threw me off, the VM must report different USB devices. The one crashing was the Apple Internal Keyboard / Trackpad, which must be surfaced differently to the VM. If you asked for 0x700a5 on Windows it would crash (as would any unknown usage).

I have tested and pushed a fix, and it has now been validated on NuGet.org so you should be able to proceed with version 1.2.8. Thank you for working with me to debug and fix the bug!

Craig

[thargy]

Having checked with usb.org (as of 08/06/2020) a5-af are currently Reserved, so looks like Apple just used them anyway :man_shrugging:.