mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

High memory issue with RunLogger #77

Open leo-schick opened 2 years ago

leo-schick commented 2 years ago

The RunLogger class caches the Output event and writes it to a local variable (node_output). There are two issues with the current implementation which might cause high memory usage during pipeline execution:

leo-schick commented 2 years ago

I noted that the ChatNotifier has a similar issue: It fees the node_output only on RunFinished, not on NodeFinished but this might be wanted. This is out of scope for this ticket.