Is there a way to deal with variable sized iterables?

[mraggi]

The problem is this line: self.total = len(gen) if total is None else total

That is executed at the beginning. But len(gen) might change throughout the iteration, but progress_bar just assumes it keeps still.

What I really want is to use a timer (so "train for 1 hour" instead of "train for so and so epochs").

import time
class Timer:
    def __init__(self, num_seconds):
        self.num_seconds = num_seconds
        self.i = None

    def __len__(self):
        if self.i is None: return 1
        eps = 1e-7
        t = time.time() - self.start
        progress = t/num_seconds
        return int((num_seconds*self.i)/(progress+eps)) # estimated size. Changes constantly.

    def __iter__(self):
        self.start = time.time()
        self.i, t = 0, 0.0
        while t <= self.num_seconds:
            t = time.time() - self.start
            self.i += 1
            yield None

With this class, if you write

for t in Timer(3.0):
    # do stuff

it runs for 3 seconds. But it doesn't work with the progress_bar :(

[sgugger]

This is not supported by the current design, no.

[mraggi]

Solved it!

class VariableSizeProgressBar(progress_bar):
    def on_update(self, val, text, interrupted=False):
        self.total = len(self.gen)
        super().on_update(val,text,interrupted)

And my timer was calculating length wrong. Here is the correct timer. This way I can tell it "train for 60 seconds".

class Timer:
    def __init__(self, num_seconds):
        self.num_seconds = num_seconds
        self.i = None

    def __len__(self):
        if self.i is None: return 1
        eps = 1e-7
        t = time.time() - self.start

        return int(self.i*self.num_seconds/t)

    def __iter__(self):
        self.start = time.time()
        self.i = 0
        t = 0.0
        while t <= self.num_seconds:
            t = time.time() - self.start
            self.i += 1
            yield None

[sgugger]

Clever trick :)