OpenFn / kit

The bits & pieces that make OpenFn work. (diagrammer, cli, compiler, runtime, runtime manager, logger, etc.)
12 stars 12 forks source link

Worker: batch log events #790

Open josephjclark opened 1 month ago

josephjclark commented 1 month ago

The worker currently sends log lines individually. Because there's a lot of logging, it can result in tens of logs per run per second.

During times of high load, that means a lot of strain coming from log events.

We should batch up subject to an argment driven limit - say every second. The worker batches log events up and only sends once per second. Probably on a throttle rather than debounce.