Undefined method '>' for nil:NilClass

[faraaz-deepsource]

I have been getting this exception from the parser, while I am parsing the source code. Unfortunately I am not able to determine on which code this happens because I don't have access to the code that ProcessedSource#from_file is running on. However it seems to be related to numparams. Could it be similar to #616 or #186?

NoMethodError: undefined method `>' for nil:NilClass
  from parser/ruby31.rb:7563:in `block in _reduce_614'
  from parser/ruby31.rb:7561:in `reverse_each'
  from parser/ruby31.rb:7561:in `_reduce_614'
  from (eval):3:in `_racc_do_parse_c'
  from (eval):3:in `do_parse'
  from parser/base.rb:190:in `parse'
  from parser/base.rb:238:in `tokenize'
  from rubocop/ast/processed_source.rb:215:in `tokenize'
  from rubocop/ast/processed_source.rb:210:in `parse'
  from rubocop/ast/processed_source.rb:38:in `initialize'
  from rubocop/ast/processed_source.rb:20:in `new'
  from rubocop/ast/processed_source.rb:20:in `from_file'
  from rubocop/runner.rb:398:in `get_processed_source'
  from rubocop/runner.rb:239:in `do_inspection_loop'
  from rubocop/runner.rb:130:in `block in file_offenses'
  from rubocop/runner.rb:155:in `file_offense_cache'
  from rubocop/runner.rb:129:in `file_offenses'
  from rubocop/runner.rb:67:in `block in warm_cache'
  from parallel.rb:587:in `call_with_index'
  from parallel.rb:557:in `process_incoming_jobs'
  from parallel.rb:537:in `block in worker'
  from parallel.rb:528:in `fork'
  from parallel.rb:528:in `worker'
  from parallel.rb:519:in `block in create_workers'
  from parallel.rb:518:in `each'
  from parallel.rb:518:in `each_with_index'
  from parallel.rb:518:in `create_workers'
  from parallel.rb:457:in `work_in_processes'
  from parallel.rb:294:in `map'
  from parallel.rb:238:in `each'
  from rubocop/runner.rb:67:in `warm_cache'
  from rubocop/runner.rb:46:in `run'
  ... <private code>

I have tried running it on different permutations of numparams in procs and lambdas but I'm not able to reproduce it

[iliabylich]

What is the version of that parser that you use? You backtrace doesn't match ruby31.rb in the master branch.

Could it be similar to https://github.com/whitequark/parser/issues/616 or https://github.com/whitequark/parser/issues/186?

I'm pretty sure it is a similar issue. Once I know what is your version I think it should be easy to find the smallest reproducible example.

[faraaz-deepsource]

@iliabylich its v3.0.2

[faraaz-deepsource]

Here's the backtrace from cloud console

[iliabylich]

Hmm, honestly I have no clue how to trigger this error. I know why it happens but I can't get exact source code to trigger it.

Basically there are two stacks that don't match:

1. stack of nested contexts (to track nested blocks/modules/classes/method definitions)
2. and a stack of maximum number of numbered parameters (to handle nested blocks with numparams)

and the error comes from a mismatch in terms of their size, looks like for some code we don't push enough data to one of them, and so when we "zip" them and iterate together one element doesn't have a match. The problem is that I don't see it from the code and I can't reproduce it.

An issue like this bothered me some time ago and so I added this helper that runs after every unit test, to make sure that all stacks are always consistent. Your code snippet is either something parser is unaware of or (which is worse) it fails because of the error recovery that is known (at least by me) to produce incorrect behaviour.

I'm sorry but I can't fix it without knowing an exact source code that triggers it. Could you add some logging to track what is the file and the source code that make this error happen?

I'm 90% sure that it is related to incorrect error recovery, when the error in source is detected (and so it is reduced into error rule and the parsing continues) but the intermediate state is not rolled back.

Also, just as an extra hint for me: what are your parser options?

[faraaz-deepsource]

What about #742 ? That seems to be giving a similar error (I apologize if it's not I'm not too familiar with this codebase) but your change for it was to check if it's not nil first.

Would that work for this issue? Also, I noticed the latest parser version has different logic for this condition than 3.0.2, Is it possible this error might go away if I upgrade?

[iliabylich]

What about https://github.com/whitequark/parser/issues/742 ? That seems to be giving a similar error (I apologize if it's not I'm not too familiar with this codebase) but your change for it was to check if it's not nil first.

No, that's a different issue.

Would that work for this issue?

No, like I mentioned above it's related to two internal stack having a different size. Swallowing an error is not a solution here.

Also, I noticed the latest parser version has different logic for this condition than 3.0.2, Is it possible this error might go away if I upgrade?

Oh, right, I changed max numparam stack to also contain information from the context stack, so I think it should be a bit more reliable on master. You could try it. And just FYI: parser always moves forward in terms of versions, and so there will be no 3.0.2.1, better always stick to the latest version if possible.

[faraaz-deepsource]

Ok I will update to the latest parser version. Will let you know if it still errors out. Thanks so much for your help! I'll close this issue for now.