Metaflow tqdm

[Adamell]

Hello, recently, I've noticed that metaflow freezes when used with tqdm and enough large number of print command in the loop. To reproduce:

from metaflow import FlowSpec, step
from tqdm import tqdm

class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.a)

    @step
    def a(self):
        for i in tqdm(range(1000000)):
            print(i)

        self.next(self.end)

    @step
    def end(self):
        print("finishing...")

if __name__ == '__main__':
    LinearFlow()

astroid==2.9.3
boto3==1.20.54
botocore==1.23.54
certifi==2021.10.8
charset-normalizer==2.0.12
click==8.0.3
idna==3.3
isort==5.10.1
jmespath==0.10.0
lazy-object-proxy==1.7.1
mccabe==0.6.1
metaflow==2.5.0
platformdirs==2.5.0
pylint==2.12.2
python-dateutil==2.8.2
requests==2.27.1
s3transfer==0.5.1
six==1.16.0
toml==0.10.2
tqdm==4.62.3
typing-extensions==4.1.1
urllib3==1.26.8
wrapt==1.13.3

Python 3.8.3

[hamelsmu]

Hi @Adamell I am not sure about tqdm, but in the meantime, you can potentially unblock yourself with fastprogress

I've had this happen to me before, and I used fast progress to get me out of this jam. I hope this helps. Example is below:

cc: @jph00 (please chime in if appropriate, I recall perhaps something along these lines being one of the motivations of creating fastprogress

from metaflow import FlowSpec, step
from fastprogress.fastprogress import progress_bar

class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.a)

    @step
    def a(self):
        for i in progress_bar(range(1000000)):
            print(i)

        self.next(self.end)

    @step
    def end(self):
        print("finishing...")

if __name__ == '__main__':
    LinearFlow()

[jis478]

@Adamell I've found that tqdm doesn't freeze the execution but fails to print its progress bar on screen. It just goes under the hood.