Canner / WrenAI

🚀 Open-source SQL AI Agent for Text-to-SQL. Make Text2SQL Easy! 🙌
https://getwren.ai/oss
GNU Affero General Public License v3.0
1.76k stars 161 forks source link

Support Spark SQL data source #708

Open andreashimin opened 1 day ago

andreashimin commented 1 day ago

Is your feature request related to a problem? Please describe. Provides users with the ability to connect and set up Spark SQL as a data source, enabling integration with distributed data processing frameworks directly within Wren AI's connection setup.

Describe the solution you'd like

For more details, see Creating a New Data Source Connector in our CONTRIBUTING.md.

Additional context Add any other context or screenshots about the feature request here.

goldmedal commented 1 day ago

The first step of this issue is implementing the SparkSQL connector on Wren engine side. Wren engine is powered by ibis-project to connect to different data sources.

Refer to ibis pyspark backend, you can follow the guide of adding DataSource to:

By the way, It would be appreciated if you could also survey how to connect to Databricks using ibis-project. 🙇