Util.GetTokensAtPosition edge cases

[MaulingMonkey]

currentToken is often null mid-line in my testing. As i understand it, a single character token will have StartIndex == StopIndex which means it can never be a current token. Whenever this happens, leftToken ends up being the last token, even if the cursor is at the start of the line, messing with when RustCompletionCommandHandler triggers intellisense.

I'm inclined to do the following refactorings:

• Add Utils.GetIdentTokenAtPosition for use in e.g. RustF1HelpCommandHandler and RustGoToDefinitioncommandHandler, which will return the ident if the cursor is in either | position in e.g. the expression: a |ident| b, (|ident|), etc. despite the left/current tokens being [whitespace, ident] in one case and [ident, whitespace] in the other.

• Replace GetTokensAtPosition with GetLineTokensBeforePosition? returning IEnumerable<IToken> or IList<IToken>, so we can still avoid triggering intellisense on e.g. let f, which requires multiple tokens of lookbehind in RustCompletionCommandHandler.

[MaulingMonkey]

My GetIdentTokenAtPosition suggestion is a bad idea - RustF1HelpCommandHandler wants to handle more than ident tokens, best to leave the filtering of what tokens the caller wants up to the caller.

That said, I am changing/replacing the behavior of GetTokensAtPosition (possibly with a rename) such that ide|nt returns the same token for both left/right (whereas foo |ident will return the whitespace and ident tokens)

[MaulingMonkey]

My take on fixing: https://github.com/MaulingMonkey/VisualRust/commit/7b9f314a6db4d54a2afff6942cdb3d758fb5a552

[MaulingMonkey]

Closing this as it's part of my pull request.