MNIST DEMO Running very slow！

intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray

https://analytics-zoo.readthedocs.io/

Apache License 2.0

17 stars 4 forks source link

MNIST DEMO Running very slow！ #610

Open FolkScientistInDL opened 4 years ago

FolkScientistInDL commented 4 years ago

hello，I am testing this code https://github.com/intel-analytics/analytics-zoo/tree/master/pyzoo/zoo/examples/pytorch/train/mnist for simple image classification. It can run successfully but its running very much slower than pure pytorch cpu implement. My cpu is E5-2678V3. Only one of my 24 CPU cores is running after I set the “cores” parameter to 12 or 24. So how to use all of the cores and how to run the network on half precision float(FP16) to accelerate the network. Hope the developer can reply me soon Thanks！

qiuxin2012 commented 4 years ago

@GavinGu07 Please have a look.

GavinGu07 commented 4 years ago

It is because you didn't set OMP_NUN_THREADS. If you hope to run with 12 or 24 cores, please set export OMP_NUN_THREADS=12 or export OMP_NUN_THREADS=24.
Since LeNet is a relatively simple model, the cost of training is less than the cost of network. So, it may take about 30% more time to train this model on Analytics Zoo than on pure pytorch. To accelerate your training and increase the CPU utilization, you should increase the batch size.
Analytics Zoo is already using FP16 to synchronize parameters, and you don't have to set it.

helenlly commented 3 years ago

@FolkScientistInDL any more questions on our answers?