search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

How can we insert tuples into a relational database? #408

Open wildboarking opened 6 years ago

wildboarking commented 6 years ago

I found most of the topics are about reading from some sources. But how to write the results to DB?

codywilbourn commented 6 years ago

Writing tuples to databases in storm would be a side-effect of a bolt in your topology.

You'd have your spout reading data, sending it some number of bolt, and one of those bolts would do the correct INSERT or UPDATE statements, depending on your database schema. Storm provides "at least once" guarantee of tuples, which means that your database writes should be prepared for sometimes attempting to re-save a tuple that had already been processed.

The idea is similar to the RedisWordCountBolt in the examples https://github.com/Parsely/streamparse/blob/master/examples/redis/src/bolts.py

Does that answer your question?