Keywords in DocStrings and DataTables break validation

[farhan5248]

• [ ] - keywords in doc string or tables like Given annotation break syntax validation

[farhan5248]

I'll take a quick look at this tomorrow to see if I can fix it otherwise I'll come back to this after reviewing the md and Prezi documentation.

[farhan5248]

I'll either need to extend the Lexer to solve this like so https://stackoverflow.com/questions/8039357/writing-a-custom-xtext-antlr-lexer-without-a-grammar-file

When doing the following, use the generated Lexer class as the base for CustomSTLexer and just change the docstring part

public class CustomSTLexer extends Lexer {

    @Override
    public void mTokens() {
      // implement lexer here
    }
}

@Override
public void configureRuntimeLexer(Binder binder) {
    binder.bind(Lexer.class)
                .annotatedWith(Names.named(LexerBindings.RUNTIME))
                .to(CustomSTLexer.class);
}

Or just use begin each line with a / and then in the test code, strip out the / if they exist.

[farhan5248]

This works for now Will update the tests later. A new bug is that doc strings can't have empty lines which is not a problem for me right now

[farhan5248]

Vertical bars also need to be escaped which can be tedious

[farhan5248]

I should that I looked into the until operator -> but it's not greedy so if you put another keyword in between, it won't have the intended effect. Perhaps using the options directly in Xtext grammar file would work?

[farhan5248]

Multiple tags need to be escaped too, pretty much any keyword

[farhan5248]

This can get pretty ugly so for now I'll use the workaround for this problem and look into the custom lexer in the future

[farhan5248]

More steps on how to make a custom lexer https://consoliii.blogspot.com/2013/04/xtext-is-incredibly-powerful-framework.html

Also I can change Statement to a terminal that isn't """ or EOL because terminals are greedy. In that case, might match a keyword or the greedy terminal rule. Two possible solutions are

1. try defining the terminals at the top so that the greedy terminal is chosen first
2. Modify the generated lexer/parser code to choose the greedy rule

[farhan5248]

If I have to customise the parser

public class BetterParser extends MylangParser {
  @Override
  protected TokenSource createLexer(CharStream stream) {
    MyLexer lexer = new MyLexer(stream);
    return lexer;
  }
}

Change my MylangRuntimeModule.java to contain this method

@Override
public Class<? extends org.eclipse.xtext.parser.IParser> bindIParser() {
     return myprj.parser.BetterParser.class ;
}

[farhan5248]

The nextToken gets tokens but they have the wrong type. This is because the mTokens switch statement looks for keywords before RULE ID etc.

One option is to change the type in nextToken to Word or String The other option is to change mTokens order of processing.

I'll change nextToken first because it's a smaller code change. If that doesn't work, I'll have to intercept the alt9 value before the switch is executed and change it there. Then with every .xtext file change, I just insert the method in there to tweak the alt9 result.

[farhan5248]

Both of these are mistaken for Scenario Outline

[farhan5248]

I'll just avoid using ScenariospaceSomething for now in my tables and titles. The original problem is fixed

[farhan5248]

Need to run mbt-transformer tests and can then close this issue