ctl-click selection was faulty, instead of checking if the element wasn't in the selection I was checking if at least one element is different.
This would lead the code to believe that the element isn't in the current selection if there is at least one different item in the selection.
Summary
ctl-click selection was faulty, instead of checking if the element wasn't in the selection I was checking if at least one element is different. This would lead the code to believe that the element isn't in the current selection if there is at least one different item in the selection.