spcl / daceml

A Data-Centric Compiler for Machine Learning
https://daceml.readthedocs.io
BSD 3-Clause "New" or "Revised" License
81 stars 15 forks source link

How to fix the ERROR: "subprocess.CalledProcessError" #126

Open HeyDavid633 opened 1 year ago

HeyDavid633 commented 1 year ago

Here is my problem while run the example/plot_cuda_mish.py
why this problem occur ? Could you please help me to fix this?

 File "/home/daceml/daceml/venv/lib/python3.8/site-packages/dace/codegen/compiler.py", line 227, in configure_and_compile
    _run_liveoutput("cmake --build . --config %s" % (Config.get('compiler', 'build_type')),
  File "/home/daceml/daceml/venv/lib/python3.8/site-packages/dace/codegen/compiler.py", line 410, in _run_liveoutput
    raise subprocess.CalledProcessError(process.returncode, command, output.getvalue())
subprocess.CalledProcessError: Command 'cmake --build . --config RelWithDebInfo' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "plot_cuda_mish.py", line 192, in <module>
    dace_output = dace_mish(dace_input)
  File "/home/daceml/daceml/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/daceml/daceml/daceml/torch/module.py", line 386, in forward
    self.function = self._initialize_sdfg(actual_inputs)
  File "/home/daceml/daceml/daceml/torch/module.py", line 354, in _initialize_sdfg
    self.compiled_function = function_generator(self, dummy_inputs)
  File "/home/daceml/daceml/daceml/torch/dispatchers/cpp_torch_extension.py", line 481, in register_and_compile_torch_extension
    compiled, handle_ptr, compiled_bwd, bwd_handle_ptr = compile_and_init_sdfgs(
  File "/home/daceml/daceml/daceml/torch/dispatchers/common.py", line 80, in compile_and_init_sdfgs
    compiled_bwd: CompiledSDFG = module.backward_sdfg.compile()
  File "/home/daceml/daceml/venv/lib/python3.8/site-packages/dace/sdfg/sdfg.py", line 2141, in compile
    shared_library = compiler.configure_and_compile(program_folder, sdfg.name)
  File "/home/daceml/daceml/venv/lib/python3.8/site-packages/dace/codegen/compiler.py", line 236, in configure_and_compile
    raise cgx.CompilationError('Compiler failure:\n' + ex.output)
dace.codegen.exceptions.CompilationError: Compiler failure:
[ 20%] Building NVCC (Device) object CMakeFiles/cuda_compile_1.dir/__/__/__/__/__/__/examples/.dacecache/DaCeMish_backward/src/cuda/cuda_compile_1_generated_DaCeMish_backward_0_cuda.cu.o
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
/home/daceml/daceml/venv/lib/python3.8/site-packages/dace/codegen/../runtime/include/dace/math.h(504): error: no operator "*=" matches these operands
            operand types are: dace::vec<float, 4U> *= const dace::vec<float, 4U>
          detected during instantiation of "T dace::math::ipow(const T &, const unsigned int &) [with T=dace::vec<float, 4U>]"
/home/daceml/daceml/examples/.dacecache/DaCeMish_backward/src/cuda/DaCeMish_backward_0_cuda.cu(90): here
1 error detected in the compilation of "/home/daceml/daceml/examples/.dacecache/DaCeMish_backward/src/cuda/DaCeMish_backward_0_cuda.cu".
CMake Error at cuda_compile_1_generated_DaCeMish_backward_0_cuda.cu.o.RelWithDebInfo.cmake:276 (message):
  Error generating file
  /home/daceml/daceml/examples/.dacecache/DaCeMish_backward/build/CMakeFiles/cuda_compile_1.dir/__/__/__/__/__/__/examples/.dacecache/DaCeMish_backward/src/cuda/./cuda_compile_1_generated_DaCeMish_backward_0_cuda.cu.o

gmake[2]: *** [CMakeFiles/DaCeMish_backward_0.dir/build.make:568: CMakeFiles/cuda_compile_1.dir/__/__/__/__/__/__/examples/.dacecache/DaCeMish_backward/src/cuda/cuda_compile_1_generated_DaCeMish_backward_0_cuda.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:84: CMakeFiles/DaCeMish_backward_0.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
tbennun commented 1 year ago

Thanks for reporting! Did you make any modifications to the code? What versions are you using (of Python, numpy, dace, daceml)?

From the error itself, it looks like there is some integer power operator that DaCe does not want to compile.

HeyDavid633 commented 1 year ago

:-) Thank you very much for your timely reply. my env-versions set as below:

I just use the make install as provided here, then just add 5 lines code to print information of the GPU_device, but not make any change of other original code So That I have no idea how to fix this problem......

Thanks for reporting! Did you make any modifications to the code? What versions are you using (of Python, numpy, dace, daceml)?

From the error itself, it looks like there is some integer power operator that DaCe does not want to compile.