"String not in language" error

[justdan96]

We have a JRuby-on-Rails application that interprets TWS definitions using RLTK that then inserts them into database tables. The application previously was using this version and has now been updated to 2.2.1.

We initially saw an error saying NameError exception occurred: undefined local variable or method 'scanner' for #<RLTK::Lexer::Environment:0x2838abe5>, which seemed to be related to these lines in lib/tws_parser/parser.rb:

    # Punctuation
    rule(/#/)                         {
      # A hash indicates a comment-line iff it is the first character on the line
      scanner.pos -= 1
      token = scanner.bol? ? nil : :HASH
      if token
        scanner.pos += 1
      else
        scanner.scan_until(/\n/)
      end
      token
    }

After replacing these lines we now receive a new error:

18-Aug-2020 21:02:17.190 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log Parsing schedules...
18-Aug-2020 21:02:17.198 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log RLTK::NotInLanguage exception occurred: String not in language.
18-Aug-2020 21:02:17.200 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log lib/tws_parser/builder.rb:72:in `process'
    app/controllers/tivoli_uploads_controller.rb:37:in `create'
   app/controllers/tivoli_uploads_controller.rb:19:in `create'

I believe the issue is in the same file as before, here is the contents of lib/tws_parser/parser.rb, lines 103 to 129:

      # A hash and then a space indicates a comment-line
      rule(/^#/)           { push_state :comment }
      rule(/\n/, :comment) { pop_state }
      rule(/./, :comment)

      # Punctuation
    rule(/,/)                         { :COMMA }
    rule(/:/)                         { :COLON }
    rule(/\./)                        { :DOT }
    rule(/@/)                         { :AT_SYM }
    rule(/\+/)                        { :PLUS }
    rule(/\-/)                        { :MINUS }

    # Text
    rule(/"(\\"|[^"])*"/) do |t|
      str = t[1..-2].gsub('\"', '"')
      str.gsub!(/%([^%]+)%/) do |m|
        (@mappings && @mappings[$1]) || m
      end
      [:STRING, str]
    end
    rule(/\([^\)]*\)/)                          { |t| [:QUALIFIER, t] }
    rule(/[A-Za-z%][A-Za-z0-9_\-\\%]*/) do |t|
      t =~ /^%(.+)%$/
      mapped = $1 && @mappings && @mappings[$1]
      [:NAME, mapped || t]
    end

I guess the issue is related to the block that assigns [:STRING, str] but I don't know what I am supposed to change it to - can you please help?

[justdan96]

I have replicated the same issue in the very latest version of RLTK - log below:

Parsing schedules...
RLTK::NotInLanguage exception occurred: String not in language.  Token info:
      Seen:
      Current: ON
      Remaining: [#<RLTK::Token:0x2f6d9d1c @value=nil, @type=:RUNCYCLE, ...
lib/tws_parser/builder.rb:76:in `process'
    app/controllers/tivoli_uploads_controller.rb:37:in `create'
    app/controllers/tivoli_uploads_controller.rb:19:in `create'
Rendered tivoli_uploads/_duplicates.html.haml (31.0ms)
Rendered shared/_settings_menu.html.haml (219.0ms)
Rendered tivoli_uploads/create.html.haml within layouts/application (437.0ms)
Completed 200 OK in 31123ms (Views: 453.0ms | ActiveRecord: 0.0ms)

[justdan96]

Sorry I am closing the issue. It turned out that the correct replacement for this code:

    # Punctuation
    rule(/#/)                         {
      # A hash indicates a comment-line iff it is the first character on the line
      scanner.pos -= 1
      token = scanner.bol? ? nil : :HASH
      if token
        scanner.pos += 1
      else
        scanner.scan_until(/\n/)
      end
      token
    }

Is actually this code:

    # A hash and then a space at the start of a line indicates a comment-line
    rule(/^# /)                       { push_state :comment }
    rule(/\n/, :comment)              { pop_state }
    rule(/./, :comment)

    # Punctuation
    rule(/#/)                         { :HASH }

It was the failure to include the :HASH definition which was causing us problems.