Support Spark SQL data source

Canner / WrenAI

🚀 An open-source SQL AI (Text-to-SQL) Agent that empowers data, product teams to chat with their data. 🤘

https://getwren.ai/oss

GNU Affero General Public License v3.0

2.03k stars 211 forks source link

Support Spark SQL data source #708

Open andreashimin opened 1 month ago

andreashimin commented 1 month ago

Is your feature request related to a problem? Please describe. Provides users with the ability to connect and set up Spark SQL as a data source, enabling integration with distributed data processing frameworks directly within Wren AI's connection setup.

Describe the solution you'd like

Add a Spark SQL connector in ibis-server
Implement Spark SQL connection handling between ibis-server and Wren UI backend.
Create a UI form for users to set up the Spark SQL data source (e.g., server address, port, authentication).

For more details, see Creating a New Data Source Connector in our CONTRIBUTING.md.

Additional context Add any other context or screenshots about the feature request here.

goldmedal commented 1 month ago

The first step of this issue is implementing the SparkSQL connector on Wren engine side. Wren engine is powered by ibis-project to connect to different data sources.

Refer to ibis pyspark backend, you can follow the guide of adding DataSource to:

define the connection info
add the connector
add test cases

By the way, It would be appreciated if you could also survey how to connect to Databricks using ibis-project. 🙇

ichuniq commented 1 month ago

Hi I'm new to this project and would like to work on this.

chilijung commented 1 month ago

@ichuniq great! Here you go!

Prototype4988 commented 2 weeks ago

@chilijung Since there hasnt been any updates for this issue, I hope you dont mind if i can have a look at it.

chilijung commented 2 weeks ago

@Prototype4988 Sure go ahead!

chilijung commented 2 weeks ago

@ichuniq Are you working on this? I noticed you submitted a PR for a typo. I think you may be looking into this issue. Please help confirm. If yes, could you also share the correct progress? Thanks.

@Prototype4988 before you start, let me double check and confirm with @ichuniq thanks.

ichuniq commented 2 weeks ago

Hi @chilijung, I’ve looked into the task but haven’t started implementing it yet due to limited capacity. However, I’ll have some free time between now and the end of October.

chilijung commented 2 weeks ago

Thank you for the update, @Prototype4988. I will find other issues for you to solve, and I've cc'd you on other ones.

@ichuniq , I look forward to your contribution, and please keep us updated!