RecursionError: maximum recursion depth exceeded

[cyw3]

Lizard 1.17.5 Python 3.7.0

When I want to update lizard 1.16.3 to lizard 1.17.5, I get a error:

$ python3 lizard.py --csv --sort cyclomatic_complexity --CCN 5 -l javascript test/ 

Traceback (most recent call last):
  File "lizard.py", line 1059, in <module>
    main()
  File "lizard.py", line 1042, in main
    warning_count = printer(result, options, schema, AllResult)
  File "/tools/lizard-1.17.5/lizard_ext/__init__.py", line 17, in print_csv
    csv_output(total_factory(list(results)), options.verbose)
  File "lizard.py", line 545, in __call__
    filename, auto_read(filename))
  File "lizard.py", line 564, in analyze_source_code
    for _ in reader(tokens, reader):
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 161, in __call__
    state(token)
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  [Previous line repeated 490 more times]
  File "/tools/lizard-1.17.5/lizard_languages/js_style_language_states.py", line 77, in _expecting_statement_or_block
    self.next(self._state_global, token)
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 28, in next
    return self(token)
  File "/tools/lizard-1.17.5/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
RecursionError: maximum recursion depth exceeded

$ python3 lizard.py --csv --sort cyclomatic_complexity --CCN 5 --working_threads 8 -l javascript test/ 

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/tools/Python-v3.7.0/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "lizard.py", line 538, in __call__
    filename, auto_read(filename))
  File "lizard.py", line 557, in analyze_source_code
    for _ in reader(tokens, reader):
  File "/tools/lizard-1.17.3/lizard_languages/code_reader.py", line 161, in __call__
    state(token)
  File "/tools/lizard-1.17.3/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  File "/tools/lizard-1.17.3/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  File "/tools/lizard-1.17.3/lizard_languages/code_reader.py", line 45, in __call__
    if self._state(token):
  [Previous line repeated 483 more times]
  File "/tools/lizard-1.17.3/lizard_languages/js_style_language_states.py", line 129, in _state_global
    super(ES6ObjectStates, self)._state_global(token)
  File "/tools/lizard-1.17.3/lizard_languages/js_style_language_states.py", line 41, in _state_global
    self.statemachine_return()
  File "/tools/lizard-1.17.3/lizard_languages/code_reader.py", line 37, in statemachine_return
    self.statemachine_before_return()
  File "/tools/lizard-1.17.3/lizard_languages/js_style_language_states.py", line 49, in statemachine_before_return
    self._pop_function_from_stack()
RecursionError: maximum recursion depth exceeded
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "lizard.py", line 1052, in <module>
    main()
  File "lizard.py", line 1035, in main
    warning_count = printer(result, options, schema, AllResult)
  File "/tools/lizard-1.17.3/lizard_ext/__init__.py", line 17, in print_csv
    csv_output(total_factory(list(results)), options.verbose)
  File "/tools/Python-v3.7.0/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
RecursionError: maximum recursion depth exceeded

And when use Lizard 1.16.3 to scan the same project, it works.

[cyw3]

the test case is the index.js of prettier@2.0.5

[terryyin]

Hi,

Sorry for the long waiting. I will be on a work break next week and will fix this then.

Terry

On 15 Jul 2020, at 4:41 PM, cyw3 notifications@github.com wrote:

the test case is the index.js of prettier@2.0.5

https://user-images.githubusercontent.com/11549103/87523771-f03cfe80-c6b9-11ea-8992-da0167b644ba.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/293#issuecomment-658632433, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGASYQHL5556ASGUPJI3R3R3VTTHANCNFSM4OWHTJ5A.

[shaikh-mansurali]

We are also facing same issue.

[jackhj000]

+1

[liiri]

The core of the problem here is that the tokenizer is expecting every open html tag, i.e <Tag> to end with a closing tag </Tag>. However, js also have generics, i.e Array<Tag> and then we quickly get into endless recursion of not having any closing tags.

Example snippet out of the referenced prettier:

{
    type: 'DebuggerStatement';
    _DebuggerStatement: void;
    end: number;
    innerComments: ?Array<Comment>;
    leadingComments: ?Array<Comment>;
    loc: {
        end: { column: number, line: number },
        start: { column: number, line: number },
    };
    start: number;
    trailingComments: ?Array<Comment>;
} 

each <Comment> block is leading us to nest into another tokenizer ... Not sure whats the best approach to resolve this.

see https://github.com/microsoft/TypeScript/issues/15713

[sthagen]

To help lizard at least survive these encounters, one may apply this crude monkey patch (or something similar):

diff --git a/lizard.py b/lizard.py
index ba8b0eb..ab74698 100755
--- a/lizard.py
+++ b/lizard.py
@@ -562,8 +562,11 @@ class FileAnalyzer(object):  # pylint: disable=R0903
         tokens = reader.generate_tokens(code)
         for processor in self.processors:
             tokens = processor(tokens, reader)
-        for _ in reader(tokens, reader):
-            pass
+        try:
+            for _ in reader(tokens, reader):
+                pass
+        except RecursionError as err:
+            print("skipped:", filename, "with reason: RecursionError (unclosed tags/scopes?)")  # or something similar ...
         return context.fileinfo

[cyw3]

Thanks!