Open artob opened 13 years ago
I would love to add an "ignore whitespace" feature to Citrus, but I'm not yet sure what the best approach would be. Ideally, there would be an ignore
directive that would be a grammar feature that could take an arbitrary rule and check for it in between successful matches.
For the time being, what I ended up using is an approach like the following:
Citrus.require(File.join(File.dirname(__FILE__), 'my_grammar'))
module MyGrammar
KEYWORDS = %w(ALL AND ANY AS ASC BY CROSS DESC DISTINCT) # etc.
TERMINALS = {
:ampersand => '&',
:asterisk => '*',
:colon => ':',
:comma => ',',
:digit => /[0-9]/,
# etc.
}
# Define grammar rules for all keywords:
KEYWORDS.each do |keyword|
rule keyword.to_sym do
all(/\b#{keyword}\b/i, zero_or_one(:separator)) do
keyword.to_sym
end
end
end
# Define grammar rules for all terminals:
TERMINALS.each do |rule_name, token|
rule rule_name do
all(token, zero_or_one(:separator)) { token.to_sym }
end
end
end
In the preceding, the referenced separator
is then a rule in the Citrus grammar definition (my_grammar.citrus
) that gobbles up any whitespace and comments, in essence tokenizing the input.
This approach works fine for my present purposes, but if it could be somehow more directly supported by Citrus core, that'd still be useful; glad to hear it's on your radar. I'm not sure, either, what the best approach to this could be, but perhaps one or another of the other packrat parsers out there might have inspiration to share?
I couldn't find this mentioned in the documentation, and the only previous discussion on this that I've found was in issue #3, so asking here...
In my experience writing Citrus grammars is very productive, except for one thing: whitespace handling. Transcribing various standard BNF grammars (SQL, SPARQL, etc) into Citrus form is presently more painful than it could be, given that every terminal needs an explicit `space` appended to it as these grammars all assume that the input has been tokenized.
To keep the production rule definitions sane, as well as to keep them consistent with those in the standard grammar being transcribed, this incentivizes workarounds like the following:
The above approach works fine, of course, but seems rather redundant and not a little laborious.
Is there by any chance a magic option I've missed somewhere that would automatically consume any trailing whitespace after recognizing a terminal? Alternatively, is there perhaps a way to feed the
#parse
method with a sequence of tokens (at its simplest, an Enumerable of strings) instead of giving it an input string?Thanks for taking the time to read this, and kudos for the awesome job you've done on Citrus so far: the documentation is superb and the source code is a pleasure to read.