Open js-truework opened 1 year ago
What is the SLY input that is causing this problem?
Note: I am more than happy to fix bugs in the project, but will not be making further package releases. People who want current code should vendor SLY into their project.
As far as I can tell, this is happening at import time from a dynamic import calling exec_module
. I don't think it's happening at any actual SLY input. But I admit that I really don't understand how metaclasses work, so I don't understand how LexerMeta.__new__()
is being called by simply running from sly import Parser
in a separate package (in this case, scim2-filter-parser is doing that import, and a recursive dynamic import is getting hung up on from .lex import *
from sly/__init__.py
.
I don't understand how Lexer._collect_rules()
is injecting the re
module into the rules, but the re
module is definitely not a string or a callable, which is why pattern
is undefined.
Maybe additional sentry output might be helpful? Sorry, this is a bug that I really don't understand well and it seems to be related to dynamic importing and metaclass evaluation, both of which are topics that I don't understand particularly well!
LexerMeta.__new__()
is runcls._build()
runs on the Lexer classOne of the rules being iterated over has a value
containing the re module.
There's definitely an improvement that can be made with regard to the if-elif
statement. It's unclear to me why this would work with a normal import and not with a dynamic import however. That will need to be investigated a bit further.
Note: I am aware that this project is no longer being maintained. I'm posting this issue here so that if others experience it, they might be able to work around it as well.
Sly's Lexer class behaves strangely when being dynamically imported, such as through
ddtrace-run
's_exec_module
. This behavior seems to stem from theclass Lexer(metaclass=LexerMeta)
.LexerMeta.__new__
seems to be called on dynamic import, presumably to construct theLexer
class. WhenLexerMeta.__new__
callscls._build()
,Lexer._build()
runs and errors out on this line:https://github.com/dabeaz/sly/blob/40009882310b6f1022ab050ff7bf6025500ee45c/src/sly/lex.py#L307
The data collected at the time of error suggests that the contents of
value
, being evaluated just above the failing line, is currently the re module, which is both not a str and not a callable, sopattern
never gets defined, which causes the titular UnboundLocalError.https://github.com/dabeaz/sly/blob/40009882310b6f1022ab050ff7bf6025500ee45c/src/sly/lex.py#L299-L307
Screenshot from sentry showing relevant information:
Workaround
The workaround that we've come up with is to manually
from sly import Lexer
prior to any of the dynamic importing happening and mark it with a noqa tag to avoid the unused import from getting removed by code quality tools. Manually importing the module first seems to fix the problem.Possible solution?
I think, if there were to be a fix in Sly, the fix would require determining what to do with
pattern
whenvalue
is neither astr
or acallable
and then having that as the default behavior. I admit that I do not understand sly very well to understand what implications having a default value here would cause, but it does seem that the lack of anelse
clause in evaluating the contents ofvalue
is the fundamental issue at play here.