Open Viewer-HX opened 2 months ago
You may want to compile with --release
to enable optimizations?
@LaurentMazare
Thanks for the suggestion. It works.
With --release
, now the training speed reaches around 8min per epoch. Improve a lot.
But I'm still confused,
(1) why the release mode is so fast? Is the optimization very aggressive?
(2)It still is slower than pytorch without GPU (around 50s per epoch.) What is the reason?
Here the training only uses CPU, is the algorithm too time-consuming?
Execution command
cargo run --example mnist-training --release --features="candle-datasets" cnn --epochs=5 --save=cnn.safetensors
Hi, I noticed an even weirder scenario, when I scale up the training epochs to 100 epochs, the training significantly decreases the accuracy.
training with GPU
Commands:
cargo run --example mnist-training --release --features="candle-datasets" --features="cuda" cnn --epochs=100 --save=../../Saved_Models/rust_cnn.safetensors
Results:
I try to use the following command to train the CNN model with mnist-training example. It works well when training linear and mlp model, but when I config it to train the CNN model, it needs approximately 1hr30mins per epoch. I think it may be abnormal.
cargo run --example mnist-training --features="candle-datasets" cnn --epochs=5 --save=cnn.safetensors
Can you please try it?