Completions not shown if the string they overlap appears again in the completion text.

[tempit]

I'm writing a language server for a CSS-based language. I have this file ( '|' marks the cursor position):

:import {
    -st-from: "./import.st.css";
    -st-named: | ;
}

Available completions at the cursor positions are 'bobo' and 'mobo' (see server response below for full completion details). Now, when I request completion at the above position, both are displayed. If I enter the char 'm', the completion 'mobo' is displayed. (good) If I enter the char 'b', no completion is displayed. I'd expect 'bobo' to be displayed.

In all 3 cases, the response from the language server is exactly the same (immediately below). Note that the label has just the string, but the newText has a leading space.

I'm also somewhat sure this worked until last week, so I suspect the move to 1.16 with the LSP changes that came with it.

[Trace - 4:59:12 PM] Received response 'textDocument/completion - (4)' in 8ms.
Result: [
    {
        "label": "mobo",
        "insertTextFormat": 2,
        "detail": "Stylable class or tag",
        "textEdit": {
            "range": {
                "start": {
                    "line": 2,
                    "character": 14
                },
                "end": {
                    "line": 2,
                    "character": 1.7976931348623157e+308
                }
            },
            "newText": " mobo"
        },
        "sortText": "a"
    },
    {
        "label": "bobo",
        "insertTextFormat": 2,
        "detail": "Stylable class or tag",
        "textEdit": {
            "range": {
                "start": {
                    "line": 2,
                    "character": 14
                },
                "end": {
                    "line": 2,
                    "character": 1.7976931348623157e+308
                }
            },
            "newText": " bobo"
        },
        "sortText": "a"
    }
]

• VSCode Version: 1.16.1
• OS Version: Ubuntu 16.04

Can anyone help?

[tempit]

Some more playing around seems to have pinpointed the issue: If the completion text contains another occurrence of the existing text, the completion is filtered out. In the above example, after typing 'm' there is no more 'm' in the string 'mobo', but after typing 'b' there is another 'b' in the remaining string. After typing 'bo', there is still another 'bo' in 'bobo', and so only when typing 'bob' the completion is shown. Easily recreated with various strings (define completion 'abcdabc' and you will only see it when reaching the d).

[tempit]

Adding proper filterText to the completions seems to have solved this. It's not clear how the filterText is used in vscode to decide which completions to display and in what order. From the functionality I've observed, it seems the filterText by default should be set to the value of insertText, not label. Is there an explanation of the filter algorithm anywhere?

[jrieken]

If I enter the char 'm', the completion 'mobo' is displayed. (good) If I enter the char 'b', no completion is displayed. I'd expect 'bobo' to be displayed.

What do you have in the buffer? mb or b? Do you type while the completions of the first request are showing or is this a fresh request?

[tempit]

@jrieken - not sure what you mean by buffer in this context. In the cases I described above, the first case is as displayed (cursor at the '|'), when triggering completions (Ctrl+Enter) The 2nd case is either when after Ctrl+Space in the above case (2 completions displayed) I type 'm', or (same behavior) when I type 'm' where the '|' is, and then Ctrl+Enter. In both cases, I see only 1 completion (which is good). The 3rd case is exactly like the second, only type in 'b' instead of 'm'. So the buffer - if I get it correctly - has 'b', not 'mb'. I should see 1 completion (bobo) but see none. [Until I added the filterText, that is]

[jrieken]

What else is on that line? Is empty otherwise? Those text edits have crazy end columns (1.7976931348623157e+308).

[tempit]

The line is shown in the first code example in the first post (it had an extra space, removed it) , here it is on it's own: (4 spaces at the start) -st-named: | ; Where '|' is the cursor, so no, it isn't empty. Since I want my completion to overwrite whatever else is on the line, I use Number.MAX_VALUE for the end position, as recommended in https://code.visualstudio.com/docs/extensions/example-language-server#_diagnostics-tips-and-tricks [Recommended for diagnostics, I figured it would work for completions, and it does]

[jrieken]

I'm also somewhat sure this worked until last week, so I suspect the move to 1.16 with the LSP changes that came with it.

cc @dbaeumer

So, checking no whitespace information got lost in translation. You want to replace all of the text after the column until the end of the line. Including the three whitespace characters, right? Basically what is marked with ^?

    -st-named:   ;
              ^^^^

Iff so, this is what happens: VS Code will take the string that is selected by that range and will match it against the label or filter text. So matching ; against m or b. This shouldn't yield a result in any case... Did I correctly understand your intent? Is there any chance I can run your code?

[tempit]

I'll try and make it clearer with images: 1st case - no characters, pressing Ctrl+Space gives labels 'bobo' and 'mobo' with insertText ' bobo' and ' mobo' respectively, both are shown.

2nd case - type 'm' and press Ctrl+Space. Same results are sent from server, only 'mobo' is shown, since - if I understood correctly, ' m' is matched against ' mobo' and ' bobo' and only the first fits. Note also the colorization is wrong on the matching letters in that completion.

3rd case - type 'b' instead of 'm', press Ctrl+Space. Same results are sent from server, but nothing is shown. I would expect 'bobo' to be shown, since ' b' is my buffer, and ' bobo'' should match.

4th case - from the 3rd case, type 'o' after the 'b', still no 'bobo'

5th case - from the 4th case, type 'b' (so the buffer is ' bob') - 'bobo' appears! This is what led me to think it has to do with repeated strings in the completion, I verified it on other strings as well...

[tempit]

And to reiterate what I said above - this all got solved as soon as I added filterText to each completion that is identical to the insertText (i.e. ' bobo' and not 'bobo' which is the label)

[jrieken]

Ok, slowing getting to the bottom of this... Since your range selects leading whitespace that whitespace is also used during compare, eg. compare b against bobo. That should fail because there is no space in the suggest label, that's why it works with the filterText and the leading white. We do some tricks when there is whitespace, will need to take a look.

[tempit]

That's more or less what I thought, thanks! The issue I'm left with here is that it seems the default filterText would be better set to the insertText (if it exists) and not the label. The reason is that the label is usually the prettified form, and the insertText actually matches the string on the line... [And the miscolorization in case 2 above]

Another thing is a description/documentation of the filtering and ranking algorithm -

1. What part of the line is compared to the completion? [Looks like it's from the last word character - not inclusive - to the cursor position. Is that right?]
2. How are completions ranked? I can mostly make mine appear first through common sense, but not always )-; I have found no docs on this.

[jrieken]

What part of the line is compared to the completion? [Looks like it's from the last word character - not inclusive - to the cursor position. Is that right?]

It is the range of the completion, so every completion item is being compared against a potentially different word and that is derived from the range of its edit. See https://github.com/Microsoft/vscode/blob/master/src/vs/editor/contrib/suggest/browser/completionModel.ts#L135

How are completions ranked? I can mostly make mine appear first through common sense, but not always )-; I have found no docs on this.

We use a fuzzy scoring algorithm that favours strong and continuous matches. Strong matches are those on upper-case character or after typical separators, like _ etc.

[jrieken]

Closing as there is no code change planned from our side.