Extending CodeReader combined symbols for bit shift operations

[benolayinka]

I'm trying to add bit shift and assign operations to the list of combined symbols in CodeReader, namely

= and <<=.

I added them to the list of combined symbols in CodeReader, but when I print the list of tokens, instances of >>= appear as '>>', '='

The regex is a bit over my head as is, do you have any suspicion why this isn't working?

[terryyin]

Have you tried to change the order of these symbols in the regex? If a “=“ is before “<<=“, the later one will never match.

On 1 Feb 2019, at 10:50 PM, Ben Olayinka notifications@github.com wrote:

I'm trying to add bit shift and assign operations to the list of combined symbols in CodeReader, namely

= and <<=.

I added them to the list of combined symbols in CodeReader, but when I print the list of tokens, instances of >>= appear as '>>', '='

The regex is a bit over my head as is, do you have any suspicion why this isn't working?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/252, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwJYmchci0hSRuEAcHUbOJb8OaLsdc3ks5vJFRRgaJpZM4aedRt.

[benolayinka]

Yep, I added >>= and <<= to the beginning of the list. I also tried removing other tokens from the list and it didn't seem to affect the tokenization. With the list unmodified, I would expect '>>=' to be tokenized as '>' and '>=' since you have '>=' as a token, but that is also not happening.

Le 2 févr. 2019 à 11:13, Terry Yin notifications@github.com a écrit :

Have you tried to change the order of these symbols in the regex? If a “=“ is before “<<=“, the later one will never match.

On 1 Feb 2019, at 10:50 PM, Ben Olayinka notifications@github.com wrote:

I'm trying to add bit shift and assign operations to the list of combined symbols in CodeReader, namely

= and <<=.

I added them to the list of combined symbols in CodeReader, but when I print the list of tokens, instances of >>= appear as '>>', '='

The regex is a bit over my head as is, do you have any suspicion why this isn't working?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/252, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwJYmchci0hSRuEAcHUbOJb8OaLsdc3ks5vJFRRgaJpZM4aedRt.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[terryyin]

Do you have a fork or somewhere I can see the code?

On 3 Feb 2019, at 3:04 AM, Ben Olayinka notifications@github.com wrote:

Yep, I added >>= and <<= to the beginning of the list. I also tried removing other tokens from the list and it didn't seem to affect the tokenization. With the list unmodified, I would expect '>>=' to be tokenized as '>' and '>=' since you have '>=' as a token, but that is also not happening.

Le 2 févr. 2019 à 11:13, Terry Yin notifications@github.com a écrit :

Have you tried to change the order of these symbols in the regex? If a “=“ is before “<<=“, the later one will never match.

On 1 Feb 2019, at 10:50 PM, Ben Olayinka notifications@github.com wrote:

I'm trying to add bit shift and assign operations to the list of combined symbols in CodeReader, namely

= and <<=.

I added them to the list of combined symbols in CodeReader, but when I print the list of tokens, instances of >>= appear as '>>', '='

The regex is a bit over my head as is, do you have any suspicion why this isn't working?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/252, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwJYmchci0hSRuEAcHUbOJb8OaLsdc3ks5vJFRRgaJpZM4aedRt.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/252#issuecomment-459990556, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwJYowbAWbR42ZxcO6wAEaAPTi1JAw9ks5vJeFbgaJpZM4aedRt.

[benolayinka]

I pushed my fork, here's the commit history. I added ">>=" as the first symbol in the "combined_symbols" list in code_reader.py.

In lizard.py, I also attach the tokens to the FileInfo object in your analyze_source_code function so I can see it in the console.

>>> i = lizard.analyze_file.analyze_source_code("AllTests.cpp", "int foo(){x>>=10")
>>> i.tokens
['int', ' ', 'foo', '(', ')', '{', 'x', '>>', '=', '10']

From code_reader.py:

combined_symbols = [">>=", "||", "&&", "===", "!==", "==", "!=", "<=",
                                ">=", "->",
                                "++", "--", '+=', '-=',
                                '*=', '/=', '^=', '&=', '|=', "..."]

[terryyin]

I tried your code, seems it works fine. Did you import your local copy of lizard or imported the globally installed one?

On 4 Feb 2019, at 5:10 PM, Ben Olayinka notifications@github.com wrote:

I pushed my fork, here's https://github.com/benolayinka/lizard/commit/4784fcda45b2299d16a7bf71edbcd8db2160e00d#diff-c34d2ca653e2b9d4862b3f159389f78bthe commit history. I added ">>=" as the first symbol in the "combined_symbols" list in code_reader.py.

In lizard.py, I also attach the tokens to the FileInfo object in your analyze_source_code function so I can see it in the console.

i = lizard.analyze_file.analyze_source_code("AllTests.cpp", "int foo(){x>>=10") i.tokens ['int', ' ', 'foo', '(', ')', '{', 'x', '>>', '=', '10'] From code_reader.py:

combined_symbols = [">>=", "||", "&&", "===", "!==", "==", "!=", "<=", ">=", "->", "++", "--", '+=', '-=', '*=', '/=', '^=', '&=', '|=', "..."] — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/terryyin/lizard/issues/252#issuecomment-460175568, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwJYruWFl9ib6U_Y6Rl-lK9-bVYEDrnks5vJ_j1gaJpZM4aedRt.

[benolayinka]

It's working for me now too. I think that reload wasn't actually reloading the module, I'm not sure why. I appreciate your help, your tool is fantastic.