cucumber-attic / gherkin2

A fast Gherkin parser in Ragel (The parser behind Cucumber)
MIT License
381 stars 220 forks source link

Make javascript parser parse BOM #100

Open aslakhellesoy opened 13 years ago

aslakhellesoy commented 13 years ago

Only one spec is failing with the js parser, and it's commented out here:

https://github.com/aslakhellesoy/gherkin/blob/master/spec/gherkin/js_lexer_spec.rb#L19

This means many files saved with a Windows editor will not parse, and this needs to be fixed. I haven't looked into the cause of the problem.

jbpros commented 10 years ago

This is biting some people using Cucumber.js: https://github.com/cucumber/cucumber-js/issues/144

simondean commented 10 years ago

Hi. The issue happens when a feature file starts with an "invisible" UTF-8 BOM (http://en.wikipedia.org/wiki/Byte_order_mark). As a BOM cannot be seen in most text editors, it makes it very difficult to diagnose. Basically only people who know what a BOM is will be able to fix the issue. I saw this issue when upgrading cucumber-js to 0.4.x. All of a sudden, a number of feature files that were previously passing, suddenly started to throw the cryptic error below.

Here's the error the issue causes:

...\node_modules\gherkin\lib\gherkin\lexer\en.js:1008
      throw new Error("Lexing error on line " + this.line_number + ": '" + content + "'. See http://wiki.github.com/cucumber/gherkin/lexingerror for more information.");
^
Error: Lexing error on line 1: 'Feature: Example feature'. See
http://wiki.github.com/cucumber/gherkin/lexingerror for more information.
    at Lexer.scan
(...\node_modules\gherkin\lib\gherkin\lexer\en.js:1008:13)
    at Object.parse
(...\node_modules\cucumber\lib\cucumber\parser.js:26:25)
    at Object.getFeatures
(...\node_modules\cucumber\lib\cucumber\runtime.js:24:35)
    at Object.start
(...\node_modules\cucumber\lib\cucumber\runtime.js:10:37)
    at Object.self._task
brasmusson commented 10 years ago

A propsed fix is in PR #301.