How to verify GPU is working?

FluxML / FluxTraining.jl

A flexible neural net training library inspired by fast.ai

https://fluxml.ai/FluxTraining.jl

MIT License

120 stars 26 forks source link

How to verify GPU is working? #47

Closed drozzy closed 3 years ago

drozzy commented 3 years ago

How can I verify that the ToGPU() callback is working as expected?

darsnack commented 3 years ago

You could check the GPU memory usage with nvidia-smi. Or check if the data is a CuArray.

drozzy commented 3 years ago

It's showing up in nvidia-smi, but the actual training seems to be taking the same amount of time. My data-loaders are pretty slow though, since they read from disk... maybe that's the reason

darsnack commented 3 years ago

Are your data loaders using DataLoaders.jl?

drozzy commented 3 years ago

Yes. GPU: 1 epoch 1:51 CPU: 1 epoch 2:44

drozzy commented 3 years ago

Oh yeah it was the file access. After I started caching stuff in memory: CPU (cached): 13s GPU (cached): 2s

darsnack commented 3 years ago

Yeah I experienced this as well, but I didn't have time to investigate. Seems like we need DataLoaders.jl to play a little better with ToGPU/CuIterator.