feedback from a noob

[max-mapper]

hi, I am new to the world of language parsers, and I took some notes while wrapping my head around nearley, perhaps they can be useful:

nearley notes

• I heard about nearley from @timoxley (@secoif on Twitter)
• wiki article on Earley parsers does not describe syntax, only algorithm
• wiki article on BNF is helpful but it's never explained in nearly why it uses a custom syntax rather than classic BNF (e.g. why it uses -> instead of ::=)
• it's unclear to me whether or not nearley syntax is more like BNF or EBNF (http://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form). it seems like a hybrid, but is definitely it's own thing
• I think a nearley -> english example like this would be helpful: http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form#Example
• IMO I think the following are vaguely worded

from calculator: "main is the nonterminal that nearley tries to parse, so we define it first."

vs from readme:

"The first nonterminal you define is the one that the parser tries to parse."

glossary

• nonterminal - basic parser constructions, made up of a name and expansions
• name - the left side of the -> in a nonterminal
• expansions - the stuff on the right side of the -> in a nonterminal. you can have many of these if they are | separated
• postprocessor - defined inside {% %} blocks at the end of production rules
• production rules - AKA 'meanings', the name for the overall expression including nonterminals and postprocessors
• id postprocessor - built in postprocessor that is a shorthand for doing function(data) {data[0];}

[YafahEdelman]

Earley is just the algorthim so the syntax is up to the specific implementation. We use -> instead of ::= just because it seems simpler and more clear, but either one is usable. In fact Nearley defines its own grammar nearley-language-bootstrapped.ne and it can be easily changed to use ::= instead of -> if some prefers ::= for their personal use. We are adding features to Nearley all the time and it supports features not in EBNF such as negation. If you'd like you can add a english example, just look at the examples we have in the example section, a simple English example would certainly be nice. hardmath123 is the person to talk too about adding a glossary, I'm sure he'll comment here soon. Its great to see more people getting interested in Nearley, I only recently started working on it myself.

[kach]

Hey, Max, that's some wonderful feedback you've got there. Thanks!

• wiki article on Earley parsers does not describe syntax, only algorithm

Well, yeah. As Jacob said, 'Earley parsers' are a breed of parsers, and the way we choose to make a syntax is independent of a parsing algorithm (as an analogy, the concept of an array is independent from whether you use Scheme's (make-vector 1 2 3) syntax or Python's [1, 2, 3] or C's {1, 2, 3}).

• wiki article on BNF is helpful but it's never explained in nearly why it uses a custom syntax rather than classic BNF (e.g. why it uses -> instead of ::=)
• it's unclear to me whether or not nearley syntax is more like BNF or EBNF (http://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form). it seems like a hybrid, but is definitely it's own thing

I accept responsibility for the weird hybrid syntax we've got in place. Part of it was that nearley started off as a toy hack, and so I didn't put any thought into the user-friendliness of the language.

Considering that as of now, nearley hasn't been used in any huge project, I'm not unwilling to make a semi-drastic backwards-incompatible change if it will strongly improve user experience. Suggestions are welcome.

• I think a nearley -> english example like this would be helpful: http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form#Example

Sure, not hard to do at all. We do have plenty of examples in the examples/ directory, if that helps.

• IMO I think the following are vaguely worded

from calculator: "main is the nonterminal that nearley tries to parse, so we define it first."

vs from readme:

"The first nonterminal you define is the one that the parser tries to parse."

Well, the meaning is the same: the first nonterminal defined in a file is the so-called 'start symbol', or the symbol you're trying to parse. How would you explain this concept?

[max-mapper]

I think the main confusing thing is the wording on the first sentence -- it makes it seems like main is special keyword. I'd rewrite main is the nonterminal that nearley tries to parse, so we define it first. as We want themainpipeline to run first so we will define it first

Most of the questions I had above were confusing, but I don't think they are wrong -- they just added cognitive load to understanding why the decisions were made. In general I don't think I disagree with everything, I just think things need more explanation.

I looked through the examples/ directory various times today and had a sort of boostrapping issue -- e.g. I didn't know enough to know what I was looking at as they grammar itself is too vague/terse to be self descriptive for me. The most helpful thing was the comments in the calculator file -- that was the example that really made a difference in my understanding.

[ghost]

I just want to chime in and say that the wording on the specific sentence Max mentioned tripped me up as well.

[kach]

I think the main confusing thing is the wording on the first sentence -- it makes it seems like main is special keyword. I'd rewrite main is the nonterminal that nearley tries to parse, so we define it first. as We want themainpipeline to run first so we will define it first

Good to know. I'll rewrite that bit.

The most helpful thing was the comments in the calculator file -- that was the example that really made a difference in my understanding.

Yes, I tried to make the calculator example the "official" tutorial grammar. Unfortunately, It's out-of-date already, so it doesn't give a good overview of the current features. I feel that if nearley is ever used in a serious project, there will be more helpful nearley examples out there, because they won't be written by me. As you can imagine, it's hard for me to write grammars from the perspective of a newbie. :-)

One of the goals of nearley was to make it super-easy to use. Good to know that that's still a challenge to be solved!

[kach]

Update: I rewrote a bit of the README, feel free to take a look and share your opinions.