lucidrains / deep-daze

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun
MIT License
4.37k stars 327 forks source link

CUDA Driver Mismatch during training #147

Open thanlon58 opened 3 years ago

thanlon58 commented 3 years ago

I was using the command line "imagine" function with --deeper and --num_layers=32 flags with a terminal window open running watch -n 0.2 nvidia-smi when suddenly (Around the 4% trained mark) the GPU information disappeared and I was only getting a message that my CUDA driver was mismatched with my library. I quit the training process, activated a tensorflow environment and ran tf.test.is_gpu_available(), which returned false.

After rebooting my machine things look like they're back to normal, but wondering if anyone knows why this happened or if it's a potentially harmful thing? Getting CUDA drivers aligned is a monumental pain so I'd like to know I'm not going to throw my whole environment out of whack by continuing to experiment with this repo.

I'm using Ubuntu 18 with a GTX 1080 Ti. I installed this repo into a fresh venv with no other installs other than updating pip.

Thanks in advance for any advice or insights! This is a super cool project and I'm excited to keep working with it.