Using multiple GPUs issue

fahadzafar commented 7 years ago

Hi,

So I'm trying to use 3 GPUs to work with neural-style-tf. The issue is, one GPU is used 100%, while the other 2 are not. This is fine, i can write a daemon to use gpu:0,1,2 but the problem is that both of the other GPUs have all their memory allocated. So the other 2 are unusable and not being used by the current execution.

Can someone help please and let me know how I can either use 3 GPUs for 3 independent tasks or all for 1 task.

Regards, Fahad

cysmith commented 7 years ago

I can look into this and some other features after December 1.

fahadzafar commented 7 years ago

Hi Cameron, that would be great thank.

On Nov 21, 2017 8:56 PM, "Cameron" notifications@github.com wrote:

I can look into this and some other features after December 1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cysmith/neural-style-tf/issues/41#issuecomment-346219488, or mute the thread https://github.com/notifications/unsubscribe-auth/AB0dCdGuIPE-Ou8pVT7d05EmyJVLdSc8ks5s4387gaJpZM4Ql8_O .

fahadzafar commented 6 years ago

Hi Cameron, any updates on this. Can you please advise me as to whats the best way to use multiple GPUs to speed up the execution. I'm okay with running 2 independent ones on 2 GPUS, in an "embarrassingly parallel way" where the 2 executions are completely independent. Currently, i cant do that since 1 run allocates the other GPU memory as mentioned above. Any help here would be greatly appreciated.

chamoulox commented 6 years ago

Hello @cysmith, @fahadzafar ,

I'm really interested by this feature, I'm currently trying to use multiple gpus and I would like to know if any of you have found something. Thanks

cysmith commented 6 years ago

To run on more than 1 gpu it seems you just do this:

for d in ['/device:GPU:2', '/device:GPU:3']:
  with tf.device(d):

So this needs to accept a list of devices instead of a single device:

parser.add_argument('--device', type=str, 
    default='/gpu:0',
    choices=['/gpu:0', '/cpu:0'],
help='GPU or CPU mode.  GPU mode requires NVIDIA CUDA. (default|recommended: %(default)s)')

Something like this: parser.add_argument('--device', type=str, default=['/gpu:0'], nargs='+')

Then call like: python neural_style.py ...blah blah blah... --device /gpu:0 /gpu:1

abhinav3 commented 6 years ago

I'm not even able to use a single GPU. Getting error as mentioned here : https://github.com/cysmith/neural-style-tf/issues/63 If any one could help with me this.

gateway commented 4 years ago

@cysmith I tried to add the command

parser.add_argument('--device', type=str, default=['/gpu:0'], nargs='+')

However when I try to run with multi gpus It fails for some reason :/

gateway@gateway-media:~/work/neural-style-tf$ python3 neural_style1.py --content_img steve-evil.jpg --content_img_dir /home/gateway/work/images --style_imgs wow.jpg --style_imgs_dir /home/gateway/work/styles/ --device /gpu:0 /gpu:1 --max_size 1000 --max_iterations 1000 --verbose;

---- RENDERING SINGLE IMAGE ----

Traceback (most recent call last):
  File "neural_style1.py", line 855, in <module>
    main()
  File "neural_style1.py", line 852, in main
    else: render_single_image()
  File "neural_style1.py", line 821, in render_single_image
    stylize(content_img, style_imgs, init_img)
  File "neural_style1.py", line 547, in stylize
    with tf.device(args.device), tf.Session() as sess:
  File "/home/gateway/anaconda3/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/gateway/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 4245, in device
    self._add_device_to_stack(device_name_or_function, offset=2)
  File "/home/gateway/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 4183, in _add_device_to_stack
    spec = _UserDeviceSpec(device_name_or_function)
  File "/home/gateway/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 101, in __init__
    self.function = pydev.merge_device(self._device_name_or_function)
  File "/home/gateway/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/device.py", line 302, in merge_device
    cached_device_spec = _cached_device_specs.get(spec, None)
TypeError: unhashable type: 'list'

cysmith / neural-style-tf

Using multiple GPUs issue #41