search

congo-cc / congo-parser-generator

The CongoCC Parser Generator, the Next Generation of JavaCC 21, which in turn was the next generation of JavaCC
https://discuss.congocc.org/
Other
37 stars 11 forks source link

Regression caused by recent f-string changes #193

Closed vsajip closed 4 months ago

vsajip commented 4 months ago

Following the updates to f-string parsing, there are some errors which appear. The source file fstr.py

f'''Hello, world!'''

produces this AST with the parser in Java:

<Module (1, 1)-(2, 1)>
  <Statement (1, 1)-(1, 21)>
    <LongSingleQuoteFString (1, 1)-(1, 20)>
      FSTRING_LONG_SINGLE_QUOTE_START: (1, 1) - (1, 4): f'''
      LONG_SINGLE_QUOTE: (1, 5) - (1, 20): '''
    Newline: (1, 21) - (1, 21): 
  Token: (3, 1) - (2, 1): EOF

As you can see, the representation of the internals of the string is incomplete, and Hello, world! doesn't appear in the dump. This is causing test failures because the C# and Python parsers don't have this problem. For example, the parser in Python gives this:

<Module (1, 1)-(2, 1)>
  <Statement (1, 1)-(1, 21)>
    <LongSingleQuoteFString (1, 1)-(1, 20)>
      FSTRING_LONG_SINGLE_QUOTE_START (1, 1)-(1, 4): f'''
      FStringText (1, 5)-(1, 17): Hello, world!
      LONG_SINGLE_QUOTE (1, 18)-(1, 20): '''
    Newline (1, 21)-(1, 21): 

  Token (3, 1)-(2, 1): EOF

Any ideas on what could be causing this?

revusky commented 4 months ago

It was the result of some mucking around in the Token.java.ftl template that I had been doing. There was a bug, which I think is now fixed. Code generation for the other languages did not have the problem because the latest changes in Token.java.ftl are not reflected (yet) in the equivalent templates.

revusky commented 4 months ago

Oh, and I mistakenly wrote "fix to #93" instead of #193 in the commit comment, so that is not shown here.

revusky commented 4 months ago

But, anyway, I think it's okay now.