search

apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.33k stars 1.2k forks source link

[DISCUSSION] More SqlLogicTest test coverage for queries, including join queries #13470

Open findepi opened 3 days ago

findepi commented 3 days ago

During SDF's upgrade to DataFusion 43 we found a bug https://github.com/apache/datafusion/issues/13425 . This was possible thanks to extensive test coverage for certain query shapes that we have internally, which is good. The bad part is that the bug could be caught at the PR stage, or before the release, should similar tests exist in DataFusion project.

Would it be useful if there were more SLT query tests, including the ones that allowed to catch https://github.com/apache/datafusion/issues/13425?

cc @alamb @schulte-lukas @gliga

alamb commented 3 days ago

Yes, 100% I think adding additional test coverage in all areas would be super helpful

We / I also regularly hit bugs that are caught by InfluxData's test suite during upgrades of DataFusion but were not caught during the DataFusion PR

Other Areas that would benefit from improved testing from my experience:

  1. Metadata handling (like attaching field level metadata, etc)
  2. UNION coverage (which we use extensively)
  3. Dictionary handling (which we also use a lot)