search

qubole / kinesis-sql

Kinesis Connector for Structured Streaming
http://www.qubole.com
Apache License 2.0
137 stars 80 forks source link

Migrate to DataSource V2 APIs #1

Open itsvikramagr opened 6 years ago

itsvikramagr commented 6 years ago

Current work is developed against SPARK 2.2.X. There are new data-source APIs in SPARK-2.3.0. We need to migrate to those APIs

krish7919 commented 6 years ago

Is there a roadmap or plan for this?

itsvikramagr commented 6 years ago

@krish7919 - We plan to complete this in next 2-3 months. if you are interested to pick it up, I would be happy to help you with design discussions and PR reviews.

saj1th commented 6 years ago

@itsvikramagr - anything i could help with, while porting to DataSource V2 - e.g Dynamodb version of MetadataCommitter ?

itsvikramagr commented 6 years ago

@saj1th - Thanks for your interest. DyanamoDb version of MetadataCommitter will anyway be useful with current data source APIs. Please go ahead and raise an issue and PR for the same.

saj1th commented 6 years ago

@itsvikramagr - Couldn't find changes related to DataSourceV2 / ContinuousReadSupport in 2.3.0 branch. Is it still a work in progress ?

itsvikramagr commented 6 years ago

@saj1th - 2.3.0 branch currently has support for micro-batch execution. Migrating it in DatasourceV2 and adding support for ContinousRead will take some time.

rodmaz commented 6 years ago

I understand the current version was tested against Spark 2.2.X, but does it also run on Spark 2.3?

itsvikramagr commented 6 years ago

Hey @rodmaz - I missed your message. It will run on spark 2.3. Current master is on 2.3.1