Open victorfuzaro opened 2 months ago
Hi @victorfuzaro , Event Hubs emulator only supports interaction with latest SDKs. The PySpark connector’s incompatibility with the emulator stems from its use of older SDKs.
To tackle such use cases, we’re actively working on Kafka support for our emulator however there is no clear ETA yet. Once Kafka support is available, we recommend using the more regularly updated native Kafka connectors. Kindly let us know if there are additional concerns/queries.
Describe the bug I am getting connection aborted/refused when trying to connect to the eventhub emulator through pyspark
To Reproduce Steps to reproduce the behavior:
Pre-requisite is to install pyspark library version: 3.5.0 or 3.5.1 from pypi with pip install pyspark==
A basic spark streaming application:
Expected behavior The pyspark application should start consuming data from the eventhub emulator streaming source continuously as it happens with a real Azure EventHub
Desktop:
Image Platform-Architecture Used:
Docker Version:
Arguments && Environment variables to start Emulator:
Emulator Launch Method:
Additional context: I can send/consume events from the emulator with a python script using the standard python EventHubConsumerClient library, but this is not my use case as I do need to consume with pyspark, so I guess something may be wrong somehow with the connectivity or maybe even with the Azure Eventhub maven project that somehow may not be compatible with the event hub emulator
Log stack: