search

antlr / antlr4-lab

A client/server for trying out and learning about ANTLR
MIT License
38 stars 11 forks source link

Official Demo Parse Question: WebIDL parse, type of all tokens is null #75

Open AntsKnows opened 1 year ago

AntsKnows commented 1 year ago

When try official demo at http://lab.antlr.org/ , select WebIDL, click run, then i can see that any WebIDL was parsed into enumValueListComma. And the type of each identifiers is null or IDENTIFIER_WEBIDL. See as: image

Then i build a local cpp-language demo:

  1. Download WebIDL.g4 from https://github.com/antlr/grammars-v4/blob/master/webidl/WebIDL.g4
  2. run Antlr4 WebIDL.g4 -Dlanguage=Cpp
  3. Use it to parse 
    interface mixin GPUObjectBase {
attribute USVString label;
};

    4.output : "line 1:0 no viable alternative at input 'interface'" or other error

AntsKnows commented 1 year ago

Add: The result is random. All input is same, but sometimes i got correct parse result, sometimes wrong result.

kaby76 commented 1 year ago

Sorry, I can't reproduce the issue for Cpp. Works fine for me.

05/19-04:27:22 ~/issues/parens/brand-new/webidl
$ trgen -t Cpp
C:\msys64\home\Kenne\issues\parens\brand-new\webidl
CSharp  WebIDL.g4 success 0.0584789
Rendering template file from Cpp/build.ps1 to Generated-Cpp/build.ps1
Rendering template file from Cpp/build.sh to Generated-Cpp/build.sh
Rendering template file from Cpp/clean.ps1 to Generated-Cpp/clean.ps1
Rendering template file from Cpp/clean.sh to Generated-Cpp/clean.sh
Rendering template file from Cpp/cmake/antlr4-generator.cmake.in to Generated-Cpp/cmake/antlr4-generator.cmake.in
Rendering template file from Cpp/cmake/antlr4-runtime.cmake.in to Generated-Cpp/cmake/antlr4-runtime.cmake.in
Rendering template file from Cpp/cmake/Antlr4Package.md to Generated-Cpp/cmake/Antlr4Package.md
Rendering template file from Cpp/cmake/ExternalAntlr4Cpp.cmake to Generated-Cpp/cmake/ExternalAntlr4Cpp.cmake
Rendering template file from Cpp/cmake/FindANTLR.cmake to Generated-Cpp/cmake/FindANTLR.cmake
Rendering template file from Cpp/cmake/README.md to Generated-Cpp/cmake/README.md
Rendering template file from Cpp/CMakeLists.txt to Generated-Cpp/CMakeLists.txt
Rendering template file from Cpp/ErrorListener.cpp to Generated-Cpp/ErrorListener.cpp
Rendering template file from Cpp/ErrorListener.h to Generated-Cpp/ErrorListener.h
Rendering template file from Cpp/makefile to Generated-Cpp/makefile
Rendering template file from Cpp/perf.sh to Generated-Cpp/perf.sh
Rendering template file from Cpp/readme.md to Generated-Cpp/readme.md
Rendering template file from Cpp/run.ps1 to Generated-Cpp/run.ps1
Rendering template file from Cpp/run.sh to Generated-Cpp/run.sh
Rendering template file from Cpp/Test.cpp to Generated-Cpp/Test.cpp
Rendering template file from Cpp/test.ps1 to Generated-Cpp/test.ps1
Rendering template file from Cpp/test.sh to Generated-Cpp/test.sh
Copying source file from C:/msys64/home/Kenne/issues/parens/brand-new/webidl/WebIDL.g4 to Generated-Cpp/WebIDL.g4
Copying source file from C:/msys64/home/Kenne/issues/parens/brand-new/webidl/desc.xml to Generated-Cpp/desc.xml
05/19-04:27:29 ~/issues/parens/brand-new/webidl
$ cd Generated-Cpp/
05/19-04:27:33 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ make
bash build.sh
CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22624.
-- The C compiler identification is MSVC 19.36.32532.0
-- The CXX compiler identification is MSVC 19.36.32532.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_static' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:129 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:20 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/ExternalProject.cmake:2254 (message):
  Policy CMP0114 is not set: ExternalProject step targets fully adopt their
  steps.  Run "cmake --help-policy CMP0114" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  ExternalProject target 'antlr4_runtime' would depend on the targets for
  step(s) 'build_shared' under policy CMP0114, but this is being left out for
  compatibility since the policy is not set.
Call Stack (most recent call first):
  cmake/ExternalAntlr4Cpp.cmake:146 (ExternalProject_Add_StepTargets)
  CMakeLists.txt:20 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found ANTLR: C:/Users/Kenne/.m2/antlr4-4.12.0-complete.jar (found version "4.12.0")
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/Kenne/issues/parens/brand-new/webidl/Generated-Cpp/build
MSBuild version 17.6.3+07e294721 for .NET Framework

  Checking Build System
  Creating directories for 'antlr4_runtime'
  Building Custom Rule C:/msys64/home/Kenne/issues/parens/brand-new/webidl/Generated-Cpp/CMakeLists.txt
  Performing download step (git clone) for 'antlr4_runtime'
  Cloning into 'antlr4_runtime'...
  HEAD is now at a2c349f9b [maven-release-plugin] prepare release 4.12.0
  Performing update step for 'antlr4_runtime'
  No patch step for 'antlr4_runtime'
  Performing configure step for 'antlr4_runtime'
  loading initial cache file c:/temp/antlr4_runtime/tmp/antlr4_runtime-cache-Release.cmake
  CMake Warning at CMakeLists.txt:12 (message):
  -- Building without demo. To enable demo build use: -DWITH_DEMO=True
    Build type not set, falling back to Release mode.

     To specify build type use:
     -DCMAKE_BUILD_TYPE=<mode> where <mode> is Debug or Release.

  -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22624.
  -- The C compiler identification is MSVC 19.36.32532.0
  -- The CXX compiler identification is MSVC 19.36.32532.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64
  /cl.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x
  64/cl.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - not found
  -- Found Threads: TRUE
  CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/FetchContent.cmake:1267 (message):
    The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
    not set.  The policy's OLD behavior will be used.  When using a URL
    download, the timestamps of extracted files should preferably be that of
    the time of extraction, otherwise code that depends on the extracted
    contents might not be rebuilt if the URL changes.  The OLD behavior
    preserves the timestamps from the archive instead, but this is usually not
    what you want.  Update your project to the NEW behavior or specify the
    DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
    robustness issue.
  Call Stack (most recent call first):
    runtime/CMakeLists.txt:61 (FetchContent_Declare)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Found Python: C:/Users/Kenne/AppData/Local/Programs/Python/Python310/python3.exe (found version "3.10.4") found components: Int
  erpreter
  -- Configuring done
  -- Generating done
  -- Build files have been written to: C:/Temp/antlr4_runtime/src/antlr4_runtime/runtime/Cpp
  No build step for 'antlr4_runtime'
  No install step for 'antlr4_runtime'
  Completed 'antlr4_runtime'
  Performing build_static step for 'antlr4_runtime'
  MSBuild version 17.6.3+07e294721 for .NET Framework

    Checking Build System
    Building Custom Rule C:/Temp/antlr4_runtime/src/antlr4_runtime/runtime/Cpp/runtime/CMakeLists.txt
    ANTLRErrorListener.cpp
    ANTLRErrorStrategy.cpp
    ANTLRFileStream.cpp
    ANTLRInputStream.cpp
    BailErrorStrategy.cpp
    BaseErrorListener.cpp
    BufferedTokenStream.cpp
    CharStream.cpp
    CommonToken.cpp
    CommonTokenFactory.cpp
    CommonTokenStream.cpp
    ConsoleErrorListener.cpp
    DefaultErrorStrategy.cpp
    DiagnosticErrorListener.cpp
    Exceptions.cpp
    FailedPredicateException.cpp
    InputMismatchException.cpp
    IntStream.cpp
    InterpreterRuleContext.cpp
    Lexer.cpp
    LexerInterpreter.cpp
    LexerNoViableAltException.cpp
    ListTokenSource.cpp
    NoViableAltException.cpp
    Parser.cpp
    ParserInterpreter.cpp
    ParserRuleContext.cpp
    ProxyErrorListener.cpp
    RecognitionException.cpp
    Recognizer.cpp
    RuleContext.cpp
    RuleContextWithAltNum.cpp
    RuntimeMetaData.cpp
    Token.cpp
    TokenSource.cpp
    TokenStream.cpp
    TokenStreamRewriter.cpp
    UnbufferedCharStream.cpp
    UnbufferedTokenStream.cpp
    Vocabulary.cpp
    WritableToken.cpp
    ATN.cpp
    ATNConfig.cpp
    ATNConfigSet.cpp
    ATNDeserializationOptions.cpp
    ATNDeserializer.cpp
    ATNSimulator.cpp
    ATNState.cpp
    ATNStateType.cpp
    ActionTransition.cpp
C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\Vocabulary.cpp(12,30): warning C4996: 'antlr4::dfa::Vocabulary::EM
PTY_VOCABULARY': Use the default constructor of Vocabulary instead. [C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\a
ntlr4_static.vcxproj] [C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\antlr4_runtime-build_static.vcxproj]
    AmbiguityInfo.cpp
    ArrayPredictionContext.cpp
    AtomTransition.cpp
    ContextSensitivityInfo.cpp
    DecisionEventInfo.cpp
    DecisionInfo.cpp
    DecisionState.cpp
    EpsilonTransition.cpp
    ErrorInfo.cpp
    LL1Analyzer.cpp
    LexerATNConfig.cpp
    LexerATNSimulator.cpp
    LexerAction.cpp
    LexerActionExecutor.cpp
    LexerChannelAction.cpp
    LexerCustomAction.cpp
    LexerIndexedCustomAction.cpp
    LexerModeAction.cpp
    LexerMoreAction.cpp
    LexerPopModeAction.cpp
    LexerPushModeAction.cpp
    LexerSkipAction.cpp
    LexerTypeAction.cpp
    LookaheadEventInfo.cpp
    NotSetTransition.cpp
    OrderedATNConfigSet.cpp
    ParseInfo.cpp
    ParserATNSimulator.cpp
    PrecedencePredicateTransition.cpp
    PredicateEvalInfo.cpp
    PredicateTransition.cpp
    PredictionContext.cpp
    PredictionContextCache.cpp
    PredictionContextMergeCache.cpp
    PredictionMode.cpp
    ProfilingATNSimulator.cpp
    RangeTransition.cpp
    RuleTransition.cpp
    SemanticContext.cpp
    SetTransition.cpp
    SingletonPredictionContext.cpp
    StarLoopbackState.cpp
    Transition.cpp
    TransitionType.cpp
    WildcardTransition.cpp
    DFA.cpp
    DFASerializer.cpp
    DFAState.cpp
    LexerDFASerializer.cpp
    Synchronization.cpp
    InterpreterDataReader.cpp
    Interval.cpp
    IntervalSet.cpp
    MurmurHash.cpp
    Predicate.cpp
    Any.cpp
    Arrays.cpp
    CPPUtils.cpp
    StringUtils.cpp
    Utf8.cpp
    ErrorNodeImpl.cpp
    IterativeParseTreeWalker.cpp
    ParseTree.cpp
    ParseTreeListener.cpp
    ParseTreeVisitor.cpp
    ParseTreeWalker.cpp
    TerminalNodeImpl.cpp
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\utility(247,16): warning C4244: 'initializ
ing': conversion from '_Ty' to '_Ty1', possible loss of data [C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\antlr4_s
tatic.vcxproj] [C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\antlr4_runtime-build_static.vcxproj]
            with
            [
                _Ty=int
            ]
            and
            [
                _Ty1=uint8_t
            ] (compiling source file C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\support\Utf8.cpp)
  C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\support\Utf8.cpp(95,20): message : see reference to function tem
  plate instantiation 'std::pair<uint8_t,uint8_t>::pair<int,const uint8_t&,0>(_Other1 &&,_Other2) noexcept' being compiled [C:\Temp\
  antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\antlr4_static.vcxproj]
            with
            [
                _Other1=int,
                _Other2=const uint8_t &
            ]
    Trees.cpp
    Chunk.cpp
    ParseTreeMatch.cpp
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\utility(247,54): warning C4244: 'initializ
ing': conversion from '_Ty' to '_Ty2', possible loss of data [C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\antlr4_s
tatic.vcxproj] [C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\antlr4_runtime-build_static.vcxproj]
            with
            [
                _Ty=int
            ]
            and
            [
                _Ty2=uint8_t
            ] (compiling source file C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\support\Utf8.cpp)
  C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\support\Utf8.cpp(95,34): message : see reference to function tem
  plate instantiation 'std::pair<uint8_t,uint8_t>::pair<const uint8_t&,int,0>(_Other1,_Other2 &&) noexcept' being compiled [C:\Temp\
  antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\antlr4_static.vcxproj]
            with
            [
                _Other1=const uint8_t &,
                _Other2=int
            ]
    ParseTreePattern.cpp
    ParseTreePatternMatcher.cpp
    RuleTagToken.cpp
    TagChunk.cpp
    TextChunk.cpp
    TokenTagToken.cpp
    XPath.cpp
    XPathElement.cpp
    XPathLexer.cpp
    XPathLexerErrorListener.cpp
    XPathRuleAnywhereElement.cpp
    XPathRuleElement.cpp
    XPathTokenAnywhereElement.cpp
    XPathTokenElement.cpp
    XPathWildcardAnywhereElement.cpp
    XPathWildcardElement.cpp
C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\src\tree\xpath\XPathLexer.cpp(166,48): warning C4100: 'context': unref
erenced formal parameter [C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\antlr4_static.vcxproj] [C:\msys64\home\Kenne
\issues\parens\brand-new\webidl\Generated-Cpp\build\antlr4_runtime-build_static.vcxproj]
    antlr4_static.vcxproj -> C:\Temp\antlr4_runtime\src\antlr4_runtime\runtime\Cpp\runtime\Release\antlr4-runtime-static.lib
  Building Custom Rule C:/msys64/home/Kenne/issues/parens/brand-new/webidl/Generated-Cpp/CMakeLists.txt
  Building WebIDL with ANTLR 4.12.0
  Building Custom Rule C:/msys64/home/Kenne/issues/parens/brand-new/webidl/Generated-Cpp/CMakeLists.txt
  Test.cpp
C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\Test.cpp(18,19): warning C4244: 'initializing': conversion from 'u
int64_t' to 'long', possible loss of data [C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\Test.vcxproj]
C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\Test.cpp(36,19): warning C4244: 'initializing': conversion from 'u
int64_t' to 'long', possible loss of data [C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\Test.vcxproj]
  ErrorListener.cpp
  WebIDLLexer.cpp
  WebIDLParser.cpp
  Generating Code...
  Test.vcxproj -> C:\msys64\home\Kenne\issues\parens\brand-new\webidl\Generated-Cpp\build\Release\Test.exe
  Building Custom Rule C:/msys64/home/Kenne/issues/parens/brand-new/webidl/Generated-Cpp/CMakeLists.txt
05/19-04:28:56 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ e x
05/19-04:29:03 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ cat x | bash run.sh
Cpp 0 stdin success 0.000538
05/19-04:29:18 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ cat x | bash run.sh -tree
(webIDL (definitions extendedAttributeList (definition (callbackOrInterfaceOrMixin interface (interfaceOrMixin (mixinRest mixin GPUObjectBase { (mixinMembers extendedAttributeList (mixinMember optionalReadOnly (attributeRest attribute (typeWithExtendedAttributes extendedAttributeList (type_ (singleType (distinguishableType (stringType USVString) null_)))) (attributeName label) ;)) mixinMembers) } ;)))) definitions) <EOF>)
Cpp 0 stdin success 0.000482
05/19-04:29:24 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ e x
05/19-04:31:55 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ dos2unix.exe x
dos2unix: converting file x to Unix format...
05/19-04:32:01 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ bash run.sh -tree x
(webIDL (definitions extendedAttributeList (definition (callbackOrInterfaceOrMixin interface (interfaceOrMixin (mixinRest mixin GPUObjectBase { (mixinMembers extendedAttributeList (mixinMember optionalReadOnly (attributeRest attribute (typeWithExtendedAttributes extendedAttributeList (type_ (singleType (distinguishableType (stringType USVString) null_)))) (attributeName label) ;)) mixinMembers) } ;)))) definitions) <EOF>)
Cpp 0 x success 0.000532
Total Time: 0.002346
05/19-04:32:11 ~/issues/parens/brand-new/webidl/Generated-Cpp
$ cat x
interface mixin GPUObjectBase {
    attribute USVString label;
};05/19-04:33:42 ~/issues/parens/brand-new/webidl/Generated-Cpp
$

And, over in lab.antlr.org, it also works fine.

download (5)

Same thing for the interpreted engine antlr-parse, which is what lab.antlr.org uses:

05/19-04:39:20 ~/issues/parens/brand-new/webidl
$ antlr4-parse WebIDL.g4 webIDL -tree Generated-Cpp/x
(webIDL:1 (definitions:1 extendedAttributeList:2 (definition:1 (callbackOrInterfaceOrMixin:2 interface (interfaceOrMixin:2 (mixinRest:1 mixin GPUObjectBase { (mixinMembers:1 extendedAttributeList:2 (mixinMember:4 optionalReadOnly:2 (attributeRest:1 attribute (typeWithExtendedAttributes:1 extendedAttributeList:2 (type_:1 (singleType:1 (distinguishableType:2 (stringType:1 USVString) null_:2)))) (attributeName:2 label) ;)) mixinMembers:2) } ;)))) definitions:2) <EOF>)
05/19-04:39:32 ~/issues/parens/brand-new/webidl

(Note, for grammars that have "actions", the interpreted Antlr engine won't work because the action is code in the target, like Java. A parser has to be generated and compiled, not interpreted. But, I recently updated the grammar file that lab.antlr.org uses, so all grammars with actions are now no longer even offered in the lab.antlr.org grammar drop-down menu.)