Open 2010YOUY01 opened 1 week ago
Thank you @2010YOUY01 Sounds like a great idea to me -- I have created a datafusion_contrib repo for this work in case you would like to put it there: https://github.com/datafusion-contrib/datafusion-sqllancer
This is the first interesting bug found: https://github.com/apache/datafusion/issues/11248:
It did not crash the DataFusion engine, instead it silently returned an incorrect result.
This logic bug is detected by NoREC
oracle explained in this issue's above example
Nice!
Is your feature request related to a problem or challenge?
I noticed an awesome SQL fuzzing framework SQLancer can be implemented on DataFusion, and it is able to detect many bugs even in PostgreSQL and SQLite
How SQLancer works in short
JDBC
to do SQL level testingsSQLancer
has 5 logic check oracles, one of them works like:Above showed consistency check generated Q1 (very likely to be optimized by predicate pushdown), and Q2(hard to be optimized), such test suit focus on correctness of the optimizer. There are 5 similar test oracles available to be implemented, those carefully designed checks make this testing framework really powerful.
Describe the solution you'd like
I plan to implement
SQLancer
onDataFusion
(starting with a specific test oralcleNoREC
which requires less engineering effort). For now, a minimal subset of SQL features is implemented: it hasn't detected any logical bug yet, just 2 bad-input bugs for some scalar functions showed up (Will share the code once it is cleaned up)If you have any features (SQL clauses / data types / specific functions) would like to be further tested, I can implement them first :)
Describe alternatives you've considered
SQLsmith
looks like another popular choice, I haven't looked into it carefully yet. But if it's only generating random SQL to test if the system will crash, thenSQLancer
should be a more comprehensive tool.Additional context
SQLancer's page have several papers/YouTube talk video recordings available