search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.3k stars 1.96k forks source link

[QUERY] Azure Schema registry + Spark Structured Streaming + Kafka/EventHub compatibility #29134

Closed scaladevspark closed 3 months ago

scaladevspark commented 2 years ago

Is there a way to use AZURE Schema Registry with Spark Structured Streaming? (not the opensource schema registry).

Ideally I would use some high level library like Abris (https://github.com/AbsaOSS/ABRiS), to read and write avro from/to Kafka with automatically registering/ reading the df schema in the schema registry. However, I have not managed to find anything at all that could enable spark to do to_avro or from_avro against an Azure Schema Registry.

There is an open PR that could add the functionality for some column types https://github.com/Azure/azure-event-hubs-spark/pull/615

There seems to be an unmainained repo that at least tried to do the from_avro part (https://github.com/arerlend/azure-schemaregistry-spark)

Anybody knows about a good way how to interact with Azure Schema Registry with Spark Structured Streaming?

joshfree commented 2 years ago

@conniey could you please follow up with @scaladevspark?

conniey commented 2 years ago

Unfortunately, I have no insights into Spark Structured Streaming. Pinging @hmlam who may have some insights.

hmlam commented 2 years ago

@yamin-msft is actually working on getting the EventHub Spark connector to work with Azure Schema Registry, which I think is what you want in your scenario. once we have a PR we would post it at https://github.com/Azure/azure-event-hubs-spark

github-actions[bot] commented 3 months ago

Hi @scaladevspark, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.