search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.8k forks source link

After the power cut, the gpu training speed on Symbol project dropped from 800+samples/s to 10 samples/s #20103

Closed amswly closed 3 years ago

amswly commented 3 years ago

Description

After the power cut, the gpu training speed on Symbol project dropped from 800+samples/s to 10 samples/s

Error Message

(Paste the complete error message. Please also include stack trace by setting environment variable DMLC_LOG_STACK_TRACE_DEPTH=100 before running your script.)

To Reproduce

(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. 2.

What have you tried to solve it?

1.I set kvstore = ‘local’ instead of 'decive', and the spread is about 600+samples/s now, but not quick as before.Does anyone has any solution for this problem. 2.

Environment

We recommend using our script for collecting the diagnostic information with the following command curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python3

Environment Information ``` # Paste the diagnose.py command output here ```
github-actions[bot] commented 3 years ago

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

amswly commented 3 years ago

well，the speed grows to 800+samples/s slowly during training