search

eladrich / pixel2style2pixel

Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework
https://eladrich.github.io/pixel2style2pixel/
MIT License
3.19k stars 570 forks source link

nvcc: build stopped: subcommand failed #137

Closed KramerKonstantin closed 3 years ago

KramerKonstantin commented 3 years ago

Hello, thank you so much for your amazing work. I'm trying to train a pSp on my own dataset and when I run a script:

python scripts/train.py \
--dataset_type=ffhq_encode \
--exp_dir=/path/to/experiment \
--workers=8 \
--batch_size=8 \
--test_batch_size=8 \
--test_workers=8 \
--val_interval=2500 \
--save_interval=5000 \
--encoder_type=GradualStyleEncoder \
--start_from_latent_avg \
--lpips_lambda=0.8 \
--l2_lambda=1 \
--id_lambda=0.1}

I get the following error:

Traceback (most recent call last):
  File "/home/kkramer/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1667, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "scripts/train.py", line 13, in <module>
    from training.coach import Coach
  File "./training/coach.py", line 18, in <module>
    from models.psp import pSp
  File "./models/psp.py", line 10, in <module>
    from models.encoders import psp_encoders
  File "./models/encoders/psp_encoders.py", line 8, in <module>
    from models.stylegan2.model import EqualLinear
  File "./models/stylegan2/model.py", line 7, in <module>
    from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
  File "./models/stylegan2/op/__init__.py", line 1, in <module>
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "./models/stylegan2/op/fused_act.py", line 9, in <module>
    fused = load(
  File "/home/kkramer/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1079, in load
    return _jit_compile(
  File "/home/kkramer/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1292, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/kkramer/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1404, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/kkramer/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++14 -c /home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o 
FAILED: fused_bias_act_kernel.cuda.o 
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++14 -c /home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o 
nvcc fatal   : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/kkramer/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp -o fused_bias_act.o 
In file included from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/c10/core/DeviceType.h:8,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/c10/core/Device.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/c10/core/Allocator.h:6,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from /home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp:1:
/home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp: In function ‘at::Tensor fused_bias_act(const at::Tensor&, const at::Tensor&, const at::Tensor&, int, int, float, float)’:
/home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp:7:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
    7 | #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                                          ^
/home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp:13:5: note: in expansion of macro ‘CHECK_CUDA’
   13 |     CHECK_CUDA(input);
      |     ^~~~~~~~~~
In file included from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/kkramer/.local/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from /home/kkramer/pixel2style2pixel/models/stylegan2/op/fused_bias_act.cpp:1:
/home/kkramer/.local/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
  303 |   DeprecatedTypeProperties & type() const {
      |                              ^~~~
ninja: build stopped: subcommand failed.

I am using ubuntu with NVIDIA GeForce RTX 3090, CUDA Version:10.1, torch version: 1.8.1+cu111. Please help.

Thank you

yuval-alaluf commented 3 years ago

Hi @KramerKonstantin Compiling ninja can be a pain sometimes. I recommend you check out some of the other issues related to the nvcc build. If the issues persist, I recommend running using our provided conda environment