[software] Workflow / Dataflow engine

[tarohi24]

after taking a look at a tutorial of Apache Beam, now I realized that it's not flexible in terms of types : https://towardsdatascience.com/hands-on-apache-beam-building-data-pipelines-in-python-6548898b66a5

[tarohi24]

How about Faust: (is a client of Kafka?)

[tarohi24]

Here is an example of Faust:

app = faust.App('myapp', broker='kafka://localhost')

# Models describe how messages are serialized:
# {"account_id": "3fae-...", amount": 3}
class Order(faust.Record):
    account_id: str
    amount: int

@app.agent(value_type=Order)
async def order(orders):
    async for order in orders:
        # process infinite stream of orders.
        print(f'Order for {order.account_id}: {order.amount}')

[tarohi24]

I realized it not good to implement a non-broker-existing model.

[tarohi24]

faust: how do I dump the table?