MQTT IO connector for Python

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.81k stars 4.23k forks source link

MQTT IO connector for Python #21060

Open damccorm opened 2 years ago

damccorm commented 2 years ago

MQTT IO connector forPython SDK

Imported from Jira BEAM-12732. Original Jira may contain additional context. Reported by: ayush1995.

gdiazdelrio commented 6 months ago

Hi. Is there any intention on following up with this idea? Or is it stale? We could definitely make use of an MQTT connector for Python or, in its defect, an External Transform that allowed us to import an unbounded PCollection to our Python pipeline. We want to use Dataflow to write to BigQuery, but our data is in MQTT instead of Pub/Sub, which is what all the documentation we found assumes. Thanks for any help you can provide

chamikaramj commented 1 month ago

@ahmedabu98 @damccorm do we have plans to include MQTT as a Python auto-generated x-lang wrapper ?

ahmedabu98 commented 1 month ago

Opened a quick PR to make this transform portable: #32385

@gdiazdelrio feel free to check and review it