Scrolling log problem when using tqdm as training process bar

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

https://clear.ml/docs

Apache License 2.0

5.42k stars 643 forks source link

Scrolling log problem when using tqdm as training process bar #1278

Open konstantinator opened 4 weeks ago

konstantinator commented 4 weeks ago

Hi!

I have a similar problem as this issue

I was using tqdm to show the training process in the command line, but when I tried to use Clearml together, some scrolling problems occur.

I'm using sdk.development.worker.console_cr_flush_period = 600 in my ~/clearml.conf

but it doesn't work for me

ainoam commented 4 weeks ago

@konstantinator Which clearml and tqdm versions are you using?

konstantinator commented 4 weeks ago

@konstantinator Which clearml and tqdm versions are you using?

HI, @ainoam! clearml==1.15.1 and tqdm==4.66.2

ainoam commented 3 weeks ago

@konstantinator Is this running manually or through a clearml-agent?

konstantinator commented 3 weeks ago

@ainoam

When this is executed manually, I see this: 2024-06-06 18 49 55

When this is executed using clearml-agent (in Docker mode), I see this:

ainoam commented 3 weeks ago

Thanks for the information @konstantinator, we'll take a look.