hrt: Cleanup destroy signal on input device

[sdilts]

The destroy listener needs to be removed, as something tries to reference it again.

• Attempt to make this easier to see by re-organizing the code a bit