search

dribnet / pixray

neural image generation
Other
402 stars 53 forks source link

Oops: runtime error: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal #13

Closed Desm0nt closed 3 years ago

Desm0nt commented 3 years ago

Hello. Last 3 days I recieve this error even on Tesla P100 in Google Colab Pro. This is not an Out of Memory error as I reduced the generation quality to draft, but the error remained. It appears on both pixray and clipit. If I disable PixelDraw, it works fine with normal VQGAN.

`iter: 0, loss: 0.914851, losses: 0.914851 /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. See the documentation of nn.Upsample for details. .format(mode)

Oops: runtime error: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal Try reducing --num-cuts to save memory RuntimeError Traceback (most recent call last)

in () 71 clipit.do_init(settings) 72 clear_output() ---> 73 clipit.do_run(settings) 6 frames /usr/local/lib/python3.7/dist-packages/diffvg-0.0.1-py3.7-linux-x86_64.egg/pydiffvg/render_pytorch.py in backward(ctx, grad_img) 707 use_prefiltering, 708 diffvg.float_ptr(eval_positions.data_ptr()), --> 709 eval_positions.shape[0]) 710 time_elapsed = time.time() - start 711 global print_timing RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal`
dribnet commented 3 years ago

Sorry about that - this happened recently when colab upgraded their default pytorch installation to a version not compatible with diffvg. We fixed all of the public notebooks - you can see the one line change here. So this should be working if you follow the direct link to all notebooks listed here but if you are using your own previously made copy of a notebook, you will need to also apply this one line change.

dribnet commented 3 years ago

I believe this is resolved now but if not feel free to re-open.