Lexer skeleton

[Nercury]

The idea here is to figure out how Lexer is going to look and work.

I placed all the modules into a single file, and when we finish the discussion I will move everything to appropriate files.

The // comments will be removed.

There are few decisions I made, quite possibly biased, so I will let you know my ideas:

First, I think it is possible to separate the Lexer from extensions completely, and initialize it with collected list of options. The list is not big: just operators, and lexer does not even care if they are binary or unary.

Second, I want to keep the iterator with the idea of "lexing on demand". That is, lexing only happens when someone (parser) requests next token. However, this is internal detail, but good to keep in mind.

I want to try and keep only the slices to original string. What to do about new lines? Well, I suggest we think about that later :)

So, basically, this is a starting point. Won't merge until we are happy with result.

[colin-kiegel]

I hope I didn't make too many comment. Awaiting your replies. ;-)

[Nercury]

Now I at looking at your colin/pr_twig_skeleton as an example, looks like we still using a bit different structure.

Notable differences: I decided not to hide lexer inside parser for now (I know, changing my mind all the time :) ). I am naming the container of lexer, parser the api. For now :/. I would like engine with setup to be separate. Any options the lexer needs should be passed by engine using Options object the lexer defines. Lexer will have no idea about engine. I don't like macros in error handling :/. I will try to use them though.

[Nercury]

Right now I implemented all lexer as iterator, but that does not prevent doing all lexing on first token request :).

Right now lexer is using SyntaxError/SyntaxErrorCode where SyntaxErrorCode contains just the message, and SyntaxError additionally has position in template. Note that here I have not used neither my At nor ErrorCode trait.

I am thinking about sharing SyntaxError between lexer and parser, therefore I have left it in api/error (instead of api/lexer/error).

[Nercury]

And then I thought about it. If I am striving for concern separation I should keep lexer errors in lexer mod.

Now lexer error can be one of two things: io::Error or twig::api::SyntaxError.

[colin-kiegel]

Cool, I will have a look (later). :-)

[colin-kiegel]

request to rename mod tokens -> mod token? (I would prefer singular nouns as module names).

[Nercury]

request to rename mod tokens -> mod token?

Yeah, that's just a detail.

[colin-kiegel]

I moved the generic error stuff to src/api/error/* (extension-dev perspective). I would like to try to separate abstract error wrappers/helpers a bit. Can we relocate the syntax error somewhere else? I think it belongs to lexer or lexing src/api/lexer/error.

Currently I imagine something like src/error.rs to define a Traced<TwigError>, where

pub enum TwigError {
    LexingError(LexingError),
    // ...
}

Where the Display + Debug traits should only add a thin layer of information - and we can still discuss and optimize the order of appearance on the screen. A user calling functions on the facade should always receive a Traced<TwigError>. If they need to destructure this, they should start with TwigError.

We can also think about re-exporting all specific errors in src/error.rs - this would be a clear separation from abstract helpers in src/api/error.rs

[colin-kiegel]

PS: Here is what I mean (from #12 src/engine/error.rs).

#[derive(Debug)]
pub enum TwigError {
    Lexer(LexerError),
    // ...
}

impl From<LexerError> for TwigError {
    fn from(err: LexerError) -> TwigError {
        TwigError::Lexer(err)
 }}

impl Error for TwigError {
    fn description(&self) -> &str {
        match *self {
            TwigError::Lexer(..) => "Twig lexer error.",
            // ...
}}}

impl Display for TwigError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        try!(write!(f, "{}", self.description()));

        match *self {
            TwigError::Lexer(ref e) => e.fmt(f),
            // ...
}}}

[Nercury]

Ok, looks great, I will do that.

[Nercury]

Continued in #21.