making a blocking remote invocation for every log message slows down execution

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

https://dagster.io

Apache License 2.0

11.66k stars 1.47k forks source link

making a blocking remote invocation for every log message slows down execution #4278

Open sryza opened 3 years ago

sryza commented 3 years ago

Consider:

Putting logging on a background thread
Batching logs

equ1n0x93 commented 9 months ago

Please prioritize this, in certain scenarios we have heavy operations in our data pipelines that reduce from 1 hour → 2 minutes when we remove logging / change log level...