parsing floats with expect state

[mulderp]

I am trying to parse floats with the expect state example. For this, I have this example lexer and a simple grammar:

//expect.jison
%lex

%s expect

%%
expect-floats        this.begin('expect');

<expect>[0-9]+"."[0-9]+      {
                    console.log( "found a float, = " + yytext );
                   }

<expect>\n           %{
            /* that's the end of the line, so
             * we need another "expect-number"
             * before we'll recognize any more
             * numbers
             */
            this.begin('INITIAL');
            %}

[0-9]+      console.log( "found an integer, = " + yytext );

"."         console.log( "found a dot" );
.           console.log('others');

/lex

%%

  program
        : expect-floats
    ;

And an example:

cat > example.exp
123.456

However, when I compile the grammar and try:

node expect.js example.exp

I get:

found an integer, = 12
found a dot
found an integer, = 324

/Users/pmu/projects/compilers/states/expect.js:100
        throw new _parseError(str, hash);
        ^
Error: Lexical error on line 1. Unrecognized text.
123.456
--------^

I would expect to see the float recognized. What would I be missing here? Thanks a lot!

[yosbelms]

I think there is a misunterstandig of what is the lexer and what the parser is, you should read the docs and the examples first.

[mulderp]

maybe I am confusing something, but it's hard for me to see. When reading about Flex/Bison examples, I saw state mentioned in the context of the lexer, e.g. like this: http://stackoverflow.com/questions/1130597/start-states-in-lex-flex - please let me know if in Jison, states are part of the parser part. Thanks!

[yosbelms]

States are part of the lexer. You need to return tokens to be consummed by the parser production rules. Example, not functional:

<expect>[0-9]+"."[0-9]+      {
                    console.log( "found a float, = " + yytext );
                    return 'FLOAT'  // return the token
                   }

...
// in the parser

program
     : FLOAT   // declare in production rules

[yosbelms]

@mulderp please let me know if the problem is already solved.

[mulderp]

thanks @yosbelms, just retried with your smaller example, but parsing 12.345 still gives a not recognized error. here is the small jison grammar:

%lex

%s expect

%%
<expect>[0-9]+"."[0-9]+   {
                    console.log( "found a float, = " + yytext );
                    return 'FLOAT'  // return the token
             }

<<EOF>>     {
             return 'EOF';
    }

/lex

// in the parser
%%

program
     : FLOAT  EOF
     ;

[yosbelms]

@mulderp To be honest, i've never used lexer states, I solve everything in parser productions. Here the following example may be close to what you are looking for.

%lex

%%
\s+                /* skip whitespaces */
[0-9]+"."[0-9]+   {
                    console.log( "found a float, = " + yytext );
                    return 'FLOAT'  // return the token
                  }

[0-9]+            return 'INT'
$                 return 'EOF'

/lex

// in the parser
%ebnf
%%

Program
    : NumberList EOF
    ;

Number
    : INT
    | FLOAT
    ;

NumberList
    : Number
    | NumberList Number
    ;

[mulderp]

found a lexer example with states here: https://github.com/greatcodeclub/markdown_parser/blob/master/tokens.jisonlex