allisonwang-db / pyspark-data-sources

Custom PySpark Data Sources
https://allisonwang-db.github.io/pyspark-data-sources/
Apache License 2.0
27 stars 4 forks source link
pyspark

pyspark-data-sources

pypi

This repository showcases custom Spark data sources built using the new Python Data Source API for the upcoming Apache Spark 4.0 release. For an in-depth understanding of the API, please refer to the API source code. Note this repo is demo only and please be aware that it is not intended for production use. Contributions and feedback are welcome to help improve the examples.

Installation

pip install pyspark-data-sources[all]

Usage

Install the pyspark 4.0 preview version: https://pypi.org/project/pyspark/4.0.0.dev1/

pip install "pyspark[connect]==4.0.0.dev1"

Or use Databricks Runtime 15.2 or above.

Try the data sources!

from pyspark_datasources.github import GithubDataSource

# Register the data source
spark.dataSource.register(GithubDataSource)

spark.read.format("github").load("apache/spark").show()

See more here: https://allisonwang-db.github.io/pyspark-data-sources/.

Contributing

We welcome and appreciate any contributions to enhance and expand the custom data sources. If you're interested in contributing:

Need help or have questions? Don't hesitate to open a new issue, and we'll do our best to assist you.

Development

poetry shell

Build docs

mkdocs serve