pschanely
commented
2 years ago Thanks for the report! Looks like we're having trouble resolving some of the type annotations here. I'll investigate this week and get back to you.
Closed barrywhart closed 2 years ago
pschanely
commented
2 years ago Thanks for the report! Looks like we're having trouble resolving some of the type annotations here. I'll investigate this week and get back to you.
barrywhart
commented
2 years ago Thanks! I've been really excited to try this project ever since I heard the podcast episode. In terms of Crosshair and SQLFluff, I'm hoping that the rules, each of which is fairly self contained, will be a good target for analysis.
pschanely
commented
2 years ago Ok! So, I spent some time digging into SQLFluff + CrossHair: it's an interesting use case! Most immediately, I've fixed the NameError issue at head. (essentially, the "TemplatedFile" import was guarded by an "if typing.TYPE_CHECKING:" condition; CrossHair has some ability to handle this, but it needed to be expanded to handle cases like this)
I'll cut a release later this week including the fix and update this issue then.
In the meantime, let me describe a little bit about what CrossHair does when analyzing BaseCondition._eval. CrossHair needs to supply a symbolic inputs to _eval, namely a RuleContext instance (and also self, which is a BaseRule). RuleContext is a sizable structure, with several instances of BaseSegment, which has many subclasses. CrossHair will recursively and nondeterministically pick subclasses to construct; it doesn't really create anything symbolic until it gets to a builtin type, like a string or integer. Because the classes you're attempting to construct are complex, you might consider limiting the amount of symbolic inputs you require. For instance, you probably don't want CrossHair to create a BaseRule via its constructor: instead, you might just check all the concrete rule instances directly; perhaps with a separate (unused) function that you just apply a contract to:
_RULES = Linter().get_ruleset()
def _check_no_exceptions_with_crosshair(segment: BaseSegment, raw_stack: List[RawSegment]):
""" post: True """
context = RuleContext(segment, (), (), (), raw_stack, None, dialect_selector("ansi"), None, None)
for (idx, rule) in enumerate(_RULES):
rule._eval(context)I obviously don't know anything about RuleContext here: the idea is that you would do whatever you need to construct a valid instance, and limit symbolic state where possible.
Another important point; check out the page on "Hints for your Classes." These hints apply not only to the classes of arguments, but also to the classes that must be supplied to those classes, etc.
You might have considered testing at a higher level - perhaps something that takes a SQL string. CrossHair supports some reasoning about the builtin re module, but since SQLFluff uses a third party regex module written in C, it won't do anything useful with it. At this level, you're likely better off using something like Atheris.
Whew! Ok, so I'm hoping that you'll continue to let me know how this goes. Having more real-world use cases to guide CrossHair is near the top of my wishlist, so this has already been super helpful.
pschanely
commented
2 years ago The original NameError issue should be fixed in v.0.0.22.
Going to continue to leave this issue open, though, for additional SQLFluff discussion if we want it.
pschanely
commented
2 years ago Closing this old issue. Hope the fix worked for you, and don't hesitate to reach out again!
Expected vs actual behavior Expected behavior: Successful run of
crosshair checkwith useful output.Actual behavior:
crosshair checkcrashes with a runtime error, "NameError".To Reproduce I'm trying to use Crosshair to detect bugs in the linting rules for SQLFluff, an open-source SQL linter/fixer/formatter written in Python. Unfortunately, it's a pretty sizable project, so the repro case involves cloning the full repo and running Crosscheck on a large portion of the code. If we're able to get this working and get useful output, it could be really helpful. The project is about 3 years old, and we are trying to stamp out bugs and reach a stable 1.0 release.
src/sqlfluff/core/rules/base.py, add apost: Truepostcondition to theBaseRule._eval()function.src/sqlfluff/core/rules/loader.py, replace theglob.glob()call with hardcoded results. Example below.crosshair check src/sqlfluff/rules.