Document Parser and Lexer

[etscrivner]

While there is some documentation in comments on the parser and lexer, it is fairly sparse and uses non-standard syntax. The lexer has little to no documentation and would benefit from having more.

The goal of this ticket would be to produce the following:

• Text file with approximate EBNF grammar of the command-line parser.
• Text file with documentation on the lexer, it's various modes and states, and any other helpful information.

Open to suggestions especially if there are additional pieces of documentation people would like or if they would prefer these to be inline comments or something else instead.

[siteshwar]

@etscrivner Thanks for taking on it. We would also appreciate any ideas/contributions to improve testing of parser and lexer.

[etscrivner]

@siteshwar Of course, I'm sure going through this process will help to stimulate some ideas there.

[siteshwar]

@etscrivner Are you still interested in working on this issue ?

[etscrivner]

@siteshwar I'd be interested in taking a crack at this still. I'm happy to start working on something in Markdown format that could live inside the repository. I'll need to brush up on the parser a bit, as I recall it's a bit convoluted.