mozilla / nunjucks

A powerful templating engine with inheritance, asynchronous control, and more (jinja2 inspired)
https://mozilla.github.io/nunjucks/
BSD 2-Clause "Simplified" License
8.52k stars 637 forks source link

Lexer's colno and lineno problems #1014

Open watzon opened 7 years ago

watzon commented 7 years ago

This is mostly a non-issue, at least for me, but I thought I'd put this here anyway. I am attempting to create a nunjucks parser in another language. While searching through the code for lexer.js and testing my code I realized that lineno and colno should start at 1 and not 0 (they're not arrays).

The other issue is that colno doesn't give the column of the start of the token, rather it gives the column that the token ends on.

As an example, passing this {{ "http://mozilla.github.io/" | urlize(10, true) | safe }} string through the lexer yields the following output:

{ type: 'variable-start', value: '{{', lineno: 0, colno: 0 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 0 }
{ type: 'string',
  value: 'http://mozilla.github.io/',
  lineno: 0,
  colno: 1 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 28 }
{ type: 'pipe', value: '|', lineno: 0, colno: 29 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 30 }
{ type: 'symbol', value: 'urlize', lineno: 0, colno: 31 }
{ type: 'left-paren', value: '(', lineno: 0, colno: 37 }
{ type: 'int', value: '10', lineno: 0, colno: 38 }
{ type: 'comma', value: ',', lineno: 0, colno: 40 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 41 }
{ type: 'boolean', value: 'true', lineno: 0, colno: 42 }
{ type: 'right-paren', value: ')', lineno: 0, colno: 46 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 47 }
{ type: 'pipe', value: '|', lineno: 0, colno: 48 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 49 }
{ type: 'symbol', value: 'safe', lineno: 0, colno: 50 }
{ type: 'whitespace', value: ' ', lineno: 0, colno: 54 }
{ type: 'variable-end', value: '}}', lineno: 0, colno: 55 }
fdintino commented 6 years ago

I would consider this a bug. Thanks for the report!