Reduce printing in tight loops

verigak / progress

Easy to use progress bars for Python

ISC License

1.41k stars 179 forks source link

Reduce printing in tight loops #85

Open lavrentivs opened 4 years ago

lavrentivs commented 4 years ago

Printing is slow, which is an issue for big tight loops (~100k items, at 100 items/second). These changes reduce the printing to, at most, twice per second by default. Setting print_dt = -1 will preserve the previous behaviour.

lavrentivs commented 4 years ago

This should also help with https://github.com/verigak/progress/issues/56

verigak commented 4 years ago

The latest commits caused a conflict to this PR. However it was incomplete as your change would only run on spinners, not bars. You will need to update writeln if you are still interested pursuing this.

lavrentivs commented 4 years ago

Updated. afaict all classes inherit from Infinite and there is no overwriting of writeln. In particular, this is the inheritance chain for a bar ChargingBar --> Bar --> Progress --> Infinite .

verigak commented 4 years ago

Right, but when you submitted your PR, bars where using writeln while spinners etc were using write. In any case everything is unified now.

However there are still a number of issues with your PR:

print_dt should not be an init argument. It should either be a class attribute, or passed in kwargs. Please follow the conventions of the code.
What's the rationale for the initialization to this? self.start_ts - self.print_dt * 2
Also please follow the Python style conventions

lavrentivs commented 4 years ago

Regarding your comments:

print_dt moved as a class attribute but allow kwargs to change it.
self._prev_write = self.start_ts - self.print_dt * 2 will force the next call to self.writeln('') to actually print.
Can you point me out to which Python/code conventions I am breaking?

I've also added changes to force that the last line is printed (ending a bar with 99% would be strange).

Thanks for taking the time reviewing this PR :)

verigak commented 4 years ago

I think it would be better to make print_dt a class attribute instead, like sma_window etc. Also I don't like the magic value of _prev_write. Why * 2 and not 1.5 or 3? I'd rather initialize it to 0 and then add a force=False argument in writeln to bypass the check when need it. Finally if you are going to keep track of _prev_line you might as well use it and not print at all if it hasn't canged (independent of print_dt)

praiskup commented 4 years ago

FWIW, not only printing is inefficient. There are floating point number miscalculations, combinded with a very short moving average window. Combined with NIH syndrome upstream, we keep this downstream patch for several years now.