diux-dev / cluster

train on AWS
75 stars 15 forks source link

ImageNet: step latency spikes #46

Open yaroslavvb opened 6 years ago

yaroslavvb commented 6 years ago

There are two slow steps each epoch -- 160ms and 200ms screenshot 2018-08-03 16 13 36

probably related to data loading https://github.com/diux-dev/cluster/blob/master/pytorch/recipes-yaroslav-va.ipynb#Step-latency-spikes cc @bearpelican