[MISC]: Analyze railroad diagram of stenotype's grammar

[a-recknagel]

A while back, I tried to gain an in-depth understanding of what exactly happens in backend/grammar.py, but never felt like I really got the hang of it. Since pyparsing now offers railroad diagrams, I thought I'd try again. From pyparsing3.0.0a2 onward (with the [diagrams] extension installed), the following can be run to generate stenotype's grammar railroad:

>>> from stenotype.backend.grammar import TYPE
>>> from pyparsing.diagram import to_railroad, railroad_to_html
>>> with open('output.html', 'w') as fp:
...    railroad = to_railroad(TYPE)
...    fp.write(railroad_to_html(railroad))
...  

Which should get us the following:

This issue can be used to discuss the current shape of the grammar.

---

Ultimately, it might be nice to extend the current grammar documentation with this graph (or partial reproductions thereof).

[a-recknagel]

Bearing in mind that I have not much of a clue, a couple of things caught my interest:

• some of the nodes appear as Unnamed x. Is it possible to give them a meaningful name?
• in the KEYWORDS node, the elements 'True' and 'False' appear twice, albeit in slightly different configurations
• the relationship between Unnamed 2 and [NAME ':'] TYPE
• the fact that the root node TYPE is so sequential. I would have imagined it to be a little more branch-y, similar to TYPE_exclude_UNION
• are nodes like '(' TYPE {',' TYPE} [',' '...'] ')', '[' TYPE ']', '{' TYPE ':' TYPE '}' and '{' TYPE '}' implicit or explicit?

pinging @maxfischer2781, in case this is interesting for you.

[maxfischer2781]

That's both nice and kind of embarrassing. Looks like re-organising could help both the parser and humans.

• The root node should definitely be more branchy.
  • All async nodes could be one group.
  • String and Bytes could be one group.
• The Unnamed x would do well with names.

The nodes '[' TYPE ']' and similar are explicitly named as such. This renders better in error messages.