search

LanguageDev / Yoakke

A collection of libraries for implementing compilers in .NET.
https://languagedev.github.io/Yoakke/
Apache License 2.0
141 stars 8 forks source link

Unsupported Regex #140

Closed fubar-coder closed 2 years ago

fubar-coder commented 2 years ago

Describe the bug

I'm unable to use the following regular expression as token:

'((\\[^\n\r])|[^\r\n\\'])*'

The build fails with the following errors:

0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(73,34): Error CS1011 : Leeres Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(73,37): Error CS1003 : Syntaxfehler. ":" erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(73,37): Error CS1010 : Zeilenvorschub in Konstante.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(73,37): Error CS1012 : Zu viele Zeichen im Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(73,41): Error CS1002 : ; erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(346,34): Error CS1011 : Leeres Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(346,37): Error CS1003 : Syntaxfehler. ":" erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(346,37): Error CS1010 : Zeilenvorschub in Konstante.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(346,37): Error CS1012 : Zu viele Zeichen im Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(346,41): Error CS1002 : ; erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(382,34): Error CS1011 : Leeres Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(382,37): Error CS1003 : Syntaxfehler. ":" erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(382,37): Error CS1010 : Zeilenvorschub in Konstante.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(382,37): Error CS1012 : Zu viele Zeichen im Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(382,41): Error CS1002 : ; erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(405,34): Error CS1011 : Leeres Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(405,37): Error CS1003 : Syntaxfehler. ":" erwartet.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(405,37): Error CS1010 : Zeilenvorschub in Konstante.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(405,37): Error CS1012 : Zu viele Zeichen im Zeichenliteral.
0>C:\Users\fubar\RiderProjects\TestDocumentSelector\TestDocumentSelector\Yoakke.SynKit.Lexer.Generator\Yoakke.SynKit.Lexer.Generator.LexerSourceGenerator\TestDocumentSelector.DocumentSelectorLexer.Generated.cs(405,41): Error CS1002 : ; erwartet.

Which libraries does it affect? Which Yoakke modules are you using that causes this problem.

Maybe either Yoakke.SynKit.Lexer.Generator or Yoakke.SynKit.Lexer.

To Reproduce Steps to reproduce the behavior:

  1. Create a TokenType
  2. Add a RegEx entry to the TokenType like this [Regex(@"'((\\[^\n\r])|[^\r\n\\'])*'")] Test,
  3. Add the lexer
  4. Try to build

Expected behavior A clear and concise description of what you expected to happen.

The regular expression seems to be find when using online Regex testers and it's very similar to the definition of Regexes.StringLiteral, so it should (IMHO) be usable.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

LPeter1997 commented 2 years ago

Thanks for reporting this issue!

Confirmed and fixed the bug. As soon as CI has done its thing, I'll push out an early nightly for you.