on_batch_end incredibly slow

bstriner / keras-tqdm

Keras integration with TQDM progress bars

MIT License

348 stars 41 forks source link

on_batch_end incredibly slow #29

Open Tom-Evers opened 5 years ago

Tom-Evers commented 5 years ago

When I run an epoch with about 2 million samples and a batch size of 1, Keras/Tensorflow finishes in about 45 minutes.

When I run that same epoch, with TQDMCallback, it takes over 13 hours.

It seems like there's logging going on during batches by default, which shouldn't be default behaviour. Not sure if that entirely explains it, though.

benp84 commented 5 years ago

Same issue here. The problem seems to be the call to np.mean(logs[metric], axis=None) inside the format_metrics function. When you're training 100 batches per second that log gets big fast.