search

stas00 / porting

Helper scripts and notes that were used while porting various nlp models
44 stars 6 forks source link

Disable extreme deepspeed logging #1

Open iliaschalkidis opened 3 years ago

iliaschalkidis commented 3 years ago

Hi @stas00,

Thanks for the deepspeed notebook example. Is there any way to disable deespeed logging?

I mean all these repetive lines, that go in between the usual logging of huggingface training script:

rank=0 time (ms) | optimizer_gradients: 51.24 | optimizer_step: 937.01 | optimizer_allgather: 64.85
[2021-03-10 10:46:04,811] [INFO] [logging.py:60:log_dist] [Rank 0] step=710, skipped=19, lr=[2.359596026758678e-05], mom=[[0.9, 0.999]]
[2021-03-10 10:46:04,811] [INFO] [timer.py:163:stop] 0/710, SamplesPerSec=174.80488942519466

Especially the optimizer info, which is printed constantly 😄

stas00 commented 3 years ago

I totally agree, @iliaschalkidis!

In general, pretty much any DeepSpeed-specific questions should go to https://github.com/microsoft/DeepSpeed - please feel free to tag me if it's related to transformers though, since most likely I need to know it too.

In this particular situation please watch for the resolution of this PR https://github.com/microsoft/DeepSpeed/pull/849

Currently, those timing prints are controlled by:

    "wall_clock_breakdown": false

in the DS config file - and it should be false by default - except it's not working right now and this PR I linked to will resolve it.

iliaschalkidis commented 3 years ago

Thanks @stas00,

I already added this in the config:

"steps_per_print": 5000,
"wall_clock_breakdown": false,

but it seems it only suppress some of the logging, not everything...

I'll keep up for updates!

stas00 commented 3 years ago

It should be good now, just for wait for the PR to be merged and use master or the upcoming new release.