search

LouieYang / deep-photo-styletransfer-tf

Tensorflow (Python API) implementation of Deep Photo Style Transfer
805 stars 186 forks source link

pycuda.driver.CompileError: nvcc preprocessing failed #13

Open Spenhouet opened 6 years ago

Spenhouet commented 6 years ago

I'm not sure if you can help on this issue or if someone using this implementation had the same problem but after running the training (for example with --style_option 1 after 1000 iterations), I get the following error: 

Traceback (most recent call last):
  File "deep_photostyle.py", line 115, in <module>
    main()
  File "deep_photostyle.py", line 84, in main
    best_ = smooth_local_affine(output_, input_, 1e-7, 3, H, W, args.f_radius, args.f_edge).transpose(1, 2, 0)
  File "C:\Users\spenh\Deep Photo Style Transfer\smooth_local_affine.py", line 332, in smooth_local_affine
    """)
  File "C:\Users\spenh\AppData\Local\Programs\Python\Python36\lib\site-packages\pycuda\compiler.py", line 291, in __init__
    arch, code, cache_dir, include_dirs)
  File "C:\Users\spenh\AppData\Local\Programs\Python\Python36\lib\site-packages\pycuda\compiler.py", line 255, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "C:\Users\spenh\AppData\Local\Programs\Python\Python36\lib\site-packages\pycuda\compiler.py", line 78, in compile_plain
    checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
  File "C:\Users\spenh\AppData\Local\Programs\Python\Python36\lib\site-packages\pycuda\compiler.py", line 55, in preprocess_source
    cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of C:\Users\spenh\AppData\Local\Temp\tmpzhinmn14.cu failed
[command: nvcc --preprocess -arch sm_61 -m64 -Ic:\users\spenh\appdata\local\programs\python\python36\lib\site-packages\pycuda\cuda C:\Users\spenh\AppData\Local\Temp\tmpzhinmn14.cu --compiler-options -EP]

I couldn't find any help with googling so maybe someone here knows what to do, to try or what this even come from?

My PC:

andrewginns commented 6 years ago

Looks like a problem with pycuda. Did you add cuda and cudnn to your environment variables as per the install guide?

Spenhouet commented 6 years ago

@andrewginns Thank you for your quick reply.

Yes, all environment variables are set:

PATH:

CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 CUDA_PATH_V8_0: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 NVCUDASAMPLES_ROOT: C:\ProgramData\NVIDIA Corporation\CUDA Samples\v8.0 NVCUDASAMPLES8_0_ROOT: C:\ProgramData\NVIDIA Corporation\CUDA Samples\v8.0

SystemErrorWang commented 6 years ago

Same problem here, use win10 64bit, gtx1070 gpu, cuda 8.0, python3.6 and tensorflow-gpu 1.3. Tried to search the internet but got nothing worth mention. Would like to know if there could be anyone who can help with this issue. Really appreciate for that.

LouieYang commented 6 years ago

Could you pass the PyCuda driver-test.py as shown in Linux and run the same command on Windows? @SystemErrorWang @Spenhouet

SystemErrorWang commented 6 years ago

@LouieYang I am sorry that I am not sure what does it mean by "pass the PyCuda driver-test.py as shown in Linux". I tried to find driver-test.py in windows and found a few, but they were not in the pycuda folder.(one in site-packages/theano, another in site-packages/numba)

SystemErrorWang commented 6 years ago

@LouieYang I also tested this code under linux(ubuntu 14.04) with default settings, but the code stopped automatically after 30 iterations without any trace-backs or error information, and generated an gray image.

LouieYang commented 6 years ago

@SystemErrorWang In Linux, there is a python file called "test_driver.py", under "pycuda-version-you-installed/test/", you can run pycuda unit test by python test_driver.py on Linux. You can check the detail in this link.

andrewginns commented 6 years ago

@Spenhouet Do you have the environment variables for CUDA_HOME, CUDA_PATH and CUDA_PATH_V8_0 set? Your error looks like a pycuda error as @LouieYang has mentioned. Can you successfully run any other CUDA based propgrams?

Alternatively if you're using windows try my Windows specific fork. It definitely works on my setup:

Spenhouet commented 6 years ago

@andrewginns Did you see my reply to your first question: https://github.com/LouieYang/deep-photo-styletransfer-tf/issues/13#issuecomment-347711755?

I listed there all the environment variables that are set in my system.

Also yes. I can run any other CUDA based program. Also as I already said in my original post, this error occur after running 1000 iterations (so obviously the first 1000 iterations are working).

marcomarchesi commented 6 years ago

It happened to me recently and I fixed in the following way, I hope it can helps someone. First of all, you should test if you can run a simple PyCuda code like this:

import pycuda.autoinit
import pycuda.driver as drv
import numpy

from pycuda.compiler import SourceModule
mod = SourceModule(
"""
__global__ void multiply_them(float *dest, float *a, float *b)
{
  const int i = threadIdx.x;
  dest[i] = a[i] * b[i];
}
"""
)

multiply_them = mod.get_function("multiply_them")

a = numpy.random.randn(400).astype(numpy.float32)
b = numpy.random.randn(400).astype(numpy.float32)

dest = numpy.zeros_like(a)
multiply_them(
        drv.Out(dest), drv.In(a), drv.In(b),
        block=(400,1,1), grid=(1,1))

print (dest-a*b)

If not, perhaps you are missing a Visual Studio compiler in the PATH, just add: C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin (if VS 14.0 is installed)

toropippi commented 4 years ago

I had the same problem.I confirmed that the PATH of CUDA and Visual Studio are set.

Windows 10 64 bit NVIDIA MX150 (Pascal) CUDA 10.1 PyCuda 2019.1.2 Python 3.6.5 (Anaconda)

and I got the following error message. [command: nvcc --preprocess -arch sm_61 -m64 -Ic:\users\xxxxx C:\Users\xxxxx\yyyyyy.cu --compiler-options -EP]

Then, I changed the compiler PATH, the compilation was successful.

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64" -> "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\bin\Hostx64\x64"

Thank you. (But, It is unknown why the error occurred even though the Visual Studio 2017 PATH was set.)

yichengggg commented 4 years ago

@toropippi Could you please let me know the processes you made to change the complier PATH?

dhkwnr97 commented 4 years ago

@yichengggg, In my case, I did apply method both @toropippi, @Spenhouet I appended Path to """ Yes, all environment variables are set:

PATH:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\libnvvp C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\extras\CUPTI\libx64 CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 CUDA_PATH_V8_0: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 NVCUDASAMPLES_ROOT: C:\ProgramData\NVIDIA Corporation\CUDA Samples\v8.0 NVCUDASAMPLES8_0_ROOT: C:\ProgramData\NVIDIA Corporation\CUDA Samples\v8.0 """ In my case, lastest version is v11.0. So I change whole thing v8.0 -> v11.0 , V8_0 -> V11_0, 8_0_ROOT -> 11_0_ROOT

and Adding below path into the Path C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\bin\Hostx64\x64 In my case, lastest version is v11.0. So I change 14.23.28105 -> 14.27.29110 And click "Yes" Lastly I reboot my computer. then I can fix this problem

In my case, I spend so much time because I don't know compiler's system environment setting problem and also VS community problem. So Plz check window's Start button -> search -> Visual Studio Installer -> 'installed' tap -> modification(the first button) -> CHECK : "desktop development by using C++" is checking

Good luck.