search

oth-datapipeline / ingestion-scripts

Repository for managing ingestion scripts
Apache License 2.0
0 stars 0 forks source link

Improve Performance of RSS Ingestion Pipeline #14

Closed lukasgraef closed 2 years ago

lukasgraef commented 2 years ago

Currently, faust-streaming can't keep up with the messages coming from kafka, resulting in lost message -> Batch Processing in faust could mitigate or solve those performance issues

lukasgraef commented 2 years ago

Error Message in Faust:

grafik

dnzxy commented 2 years ago

@lukasgraef maybe add a _createdAt field for each event when inserting into mongo, so that we can verify when events where pushed into mongo.