apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

https://kyuubi.apache.org/

Apache License 2.0

2.11k stars 915 forks source link

Prevent triggering spark job when executing the initialization sql #6789

Closed turboFei closed 2 weeks ago

turboFei commented 3 weeks ago

:mag: Description

Issue References 🔗

I found that, the spark.sql("show databases").isEmpty will trigger the job with community spark-3.5.2（3.1 does not).

Describe Your Solution 🔧

Using dataFrame.take(1).isEmpty instead of dataFrame.isEmpty.

Types of changes :bookmark:

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Behavior With This Pull Request :tada:

Related Unit Tests

No job triggered for take(1).isEmpty.

Checklist 📝

[ ] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

turboFei commented 2 weeks ago

Prefer to backport the spark PR to managed spark