search

CoatiSoftware / Sourcetrail

Sourcetrail - free and open-source interactive source explorer
https://www.sourcetrail.com/
GNU General Public License v3.0
14.94k stars 1.41k forks source link

Unexpected token #1102

Open ukos-git opened 4 years ago

ukos-git commented 4 years ago

I get a lot of errors like Unexpected token of type "OP" encountered at valid (but uncommonly strange-looking) python code. The code is formatted PEP compliant.

        map(
            lambda (x, y): '{:s} = {:.3f}um'.format(
                x, y * 1e6
            ), dict(
                {axis: position[axis] for axis in axes}
            ).iteritems()
        )

image image

I suspect this is an issue with the tokenizer used by SourceTrail that somehow misses to detect the opening and closing brackets correctly.

I hope the above example suffices as a mwe.

mlangkabel commented 4 years ago

This looks like a parser issue. I'm not an expert, but are you really supposed to use ( and ) after the lambda keyword?

ukos-git commented 4 years ago

I missed to mention that this is python2 code.

I would expect that unparenthesized tuples make it harder for your tokenizer to recognize operator precedence.

I am also not sure if varargslist to lambda would actually conflict with the function parameter argslist of map. So that the function parameters actually have precedence. However, lambdadef also waits for a colon after varargslist, so I guess the brackets in fpdef are always optional.

https://docs.python.org/2/reference/grammar.html parameters: '(' [varargslist] ')' varargslist: ((fpdef ['=' test] ',') ('' NAME [',' '' NAME] | '' NAME) | fpdef ['=' test] (',' fpdef ['=' test]) [',']) fpdef: NAME | '(' fplist ')' fplist: fpdef (',' fpdef) [','] ... lambdef: 'lambda' [varargslist] ':' test

Nevertheless, as I mentioned before, the above python code is valid.