search

funkelab / daisy

Block-wise task scheduling for large nD volumes.
MIT License
26 stars 17 forks source link

Tqdm Progress bar can become too big #57

Open mzouink opened 3 months ago

mzouink commented 3 months ago

When submitting a cluster task tqdm save a new line for twice for each block which can become a gigabytes file

Is there a way to make an option that set update frequency ? or use an other option that saves only the last line

pattonw commented 1 month ago

This sounds like it would be a great addition. We have an observer pattern implemented for handling updates while blockwise processing here: https://github.com/funkelab/daisy/blob/master/daisy/cl_monitor.py

I think we could consider a couple reasonable approaches: 1) add verbosity arguments to the existing monitor 2) add a new ServerObserver subclass specifically for remote execution where you do not want the progress bar at all but rather just a tidy log file