search

tree-sitter / tree-sitter-go

Go grammar for tree-sitter
MIT License
310 stars 65 forks source link

"source_file" allows "BLANK" to follow "_top_level_declaration" #46

Closed berchn closed 1 year ago

berchn commented 3 years ago

source_file allows BLANK to follow _top_level_declaration in src/grammar.json. This is resulting in an AST that doesn't appear to be valid. The following code parses without error or the addition of any zero-width tokens, but it does not compile:

package a package b package c package d

This is due to the optional(terminator) in the relevant section of the source_file rule in grammar.js. Is this intended?

adonovan commented 2 years ago

The optionality should be removed here, but I think it might require additional handling to ensure that EOF inserts a final newline (terminator): https://github.com/tree-sitter/tree-sitter-go/blob/07d722831382a043b16547b6d9202f3da07f3cb3/grammar.js#L107

I suspect in practice it's less trouble to make the parsers overgenerous in what they accept than to have to analyze the dark corners perfectly (where compilers often have bugs too).