Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
7.8k stars 738 forks source link

feature: stream logs in chunks #1856

Closed saikonen closed 4 weeks ago

saikonen commented 1 month ago

attempts to read logs with a fixed byte offset instead of all at once. Motivation is to decrease memory consumption when handling massive log files.

saikonen commented 4 weeks ago

closing this as unnecessary for the planned metaflow-service feature