fill_subvolume_with_model overwrites CUDA_VISIBLE_DEVICES

aschampion / diluvian

Flood filling networks for segmenting electron microscopy of neural tissue

MIT License

23 stars 13 forks source link

fill_subvolume_with_model overwrites CUDA_VISIBLE_DEVICES #11

Closed braingram closed 7 years ago

braingram commented 7 years ago

I'm running your code on a machine with 2 gpus: 470 and Titan Black I was able to run "diluvian train" after setting CUDA_VISIBLE_DEVICES (to ignore the 470). However, "diluvian fill" would always end up running on the CPU because CUDA_VISIBLE_DEVICES was being overwritten and the 470 isn't suitable.

I think checking if CUDA_VISIBLE_DEVICES is in os.environ before setting should fix this.

Let me know if more information is helpful.

aschampion commented 7 years ago

Thanks for the report, @braingram. The latest commit should work around this issue for now; if not, please reopen.

Device handling and the filling code is a mess right now, but with this change if you have CUDA_VISIBLE_DEVICES set it won't be overridden and the filling will fall back to using a single worker.