Erroneous completions when the cursor is not at the end of the line

[allanrenucci]

Let's consider the following example:

> {List.ran}
           ^

Let's say I have a single candidate Candidate(value="ge", displ="range", displ=false), when I tab-complete, the candidate is appended after the closing brace:

> {List.ran}ge
              ^

Maybe related to #251, although this can be replicated with any character, not only space and braces.

Using JLine 3.7.0

[gnodet]

Could you explain how to do it with the jline demo ? I've tried a bit with no success. Or even better, could you set up a unit test that could be integrated in the build ?

[allanrenucci]

I can reproduce with something like:

class JLineTerminal {
  String readLine() {
    Terminal terminal = TerminalBuilder.terminal();
    LineReader lineReader = LineReaderBuilder.builder()
      .terminal(terminal)
      .completer(new Completer())
      .parser(new Parser())
      .build();

    return lineReader.readLine(">");
  }

  static class Parser extends reader.Parser {
    static class DummyParsedLine extends ParsedLine {
      int cursor;
      String line;
      DummyParsedLine(int cursor, String line) {
        this.cursor = cursor;
        this.line = line;
      }

      @Override int cursor() { return cursor; }
      @Override String line() { return line; }

      // using dummy values, not sure what they are used for
      @Override String word() { return ""; }
      @Override int wordCursor() { return -1; }
      @Override int wordIndex() { return -1; }
      @Override List<String> words() { return Collections.emptyList(); }
    }

    @Override
    ParsedLine parse(String line, int cursor, ParseContext context) {
      return new DummyParsedLine(cursor, line);
    }
  }

  static class Completer extends reader.Completer {
    @Override
    void complete(LineReader reader, ParsedLine line, List<Candidate> candidates) {
      candidates.add(new Candidate("range"));
    }
  }

  public static void main(String[] args) {
    new JLineTerminal().readLine();
  }
}

>{}
  ^
>{}range
         ^

But I realise it might be because I did not correctly implement the Parser interface

[gnodet]

Yes, that's certainly related to your Parser implementation, as it's used to detect word boundaries and select completion candidates : in short, the completion behavior heavily depends on the Parser implementation for non trivial use cases.

[allanrenucci]

Let's say I want to complete the following buffer:

> {List.}
        ^

What should be the values in ParsedLine such that the completions are inserted after the . and not the }?

[gnodet]

It can be whatever you want, but it needs to be coherent with the Candidates returned by the completer. The Candidate value must match a work in the ParsedLine. So what you need depend on the syntax, without knowing it, I can't give a good advice. But if the { and } are separators, then the words should be {, List. and } and the candidate value can be List.foo. If it should be considered a single word, then the list of words should be {List.} and the candidate value should be {List.foo}.

[allanrenucci]

Thanks for the quick reply.

So what you need depend on the syntax, without knowing it, I can't give a good advice.

I am using JLine to implement a REPL for the Scala programming language. So { and } are separators, List is an identifier and the . let me select the members of List

[gnodet]

Then the first thing to implement is correct Parser for the Scala language. Each token in the language should be its own word so that the completion will be easier. The Parser is also used in a few other places, for example if you hit { then <enter>, the LineReader should open a new line because the Parser will indicate that the line is not correctly finished. Same for quotes...

Once you have a Parser, you can leverage it in your Completers, in particular, the ParsedLine can be of a specific type so that your completers can look into the parsed tokens and have the full context.

I've never implemented or used (programmatically) a full language auto-completion system, so I can't help much on that side, but I'm quite sure there are already parsers and completion systems in open source editors, so may be able to rely on those libraries.

[allanrenucci]

Fortunately for me the REPL will be part of the Scala compiler and I can reuse the existing parser and completion API. I just need to write the glue code that connects the compiler with the JLine API. Here is my current prototype (~150 LOC). Multi-line editing, syntax highlighting, and basic auto-completion already work well. Just need to figure out how to make completion work when the cursor is not at the end of the buffer.

I tried having a word for each token as you suggested. So given the example above, at the time I request a completion, the returned ParsedLine is:

cursor     = 6
line       = "{List.}"
word       = "}"
wordCursor = 0
wordIndex  = 3
words      = List("{", "List", ".", "}")

Completing now eats the closing brace

> {List.range
             ^

My only candidate is:

Candidate(
  /* value    = */ "range",
  /* displ    = */ "range",
  /* group    = */ null,
  /* descr    = */ null,
  /* suffix   = */ null,
  /* key      = */ null,
  /* complete = */ false
)

[gnodet]

Sorry, I missed you reply.

So the problem is that your parser returns } as the line's current word, which means that it's the word that is being completed. In order for the completion to work, your parser needs to return a dummy empty word that will be what is being completed.

The below test works well:

    @Test
    public void testComplete() throws IOException {
        reader.setCompleter((reader, line, candidates) -> candidates.add(new Candidate(
                /* value    = */ "range",
                /* displ    = */ "range",
                /* group    = */ null,
                /* descr    = */ null,
                /* suffix   = */ null,
                /* key      = */ null,
                /* complete = */ false)));
        reader.setParser((line, cursor, context) -> new ParsedLine() {
            @Override
            public String word() {
                return "";
            }
            @Override
            public int wordCursor() {
                return 0;
            }
            @Override
            public int wordIndex() {
                return 3;
            }
            @Override
            public List<String> words() {
                return Arrays.asList("{", "List", ".", "", "}");
            }
            @Override
            public String line() {
                return "{List.}";
            }
            @Override
            public int cursor() {
                return 6;
            }
        });

        assertBuffer("{List.range}", new TestBuffer("{List.}").left().tab());
    }