Store stationary sensors signals from RabbitMQ to MongoDB

[ceefour]

• Use Python 3.6 (Anaconda)
• Use pyspark
• Use Spark Structured Streaming (preferred), or at least Spark Streaming (DStream)
• RabbitMQ Spark Streaming Source, choices: (note: Spark Streaming docs say that custom receivers aren't supported for Python as of Spark 2.2, however https://github.com/radanalyticsio/streaming-amqp has a Python example, so there is hope)
  • https://github.com/Stratio/spark-rabbitmq
  • https://github.com/radanalyticsio/streaming-amqp
  • http://bahir.apache.org/docs/spark/current/spark-streaming-mqtt/
• MongoDB Spark Streaming Sink: https://docs.mongodb.com/spark-connector/master/scala/streaming/