Closed justanotheruser closed 3 years ago
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.
Please try using the settings in https://github.com/apache/incubator-mxnet/blob/master/ci/build_windows.py
@leezu Will (should) it work if I add USE_CPP_PACKAGE=1 in that script? Thanks.
Let's try that. Would you like to open a PR targeting the v1.x
branch that enables USE_CPP_PACKAGE=1
in above file? Then CI will test the USE_CPP_PACKAGE=1
build and answer the question
I tried to build with settings in https://github.com/apache/incubator-mxnet/blob/master/ci/build_windows.py
This is exact command that I used: python build_windows.py --flavour WIN_GPU_MKLDNN
Build log (start)
(venv) D:\incubator-mxnet\ci>python build_windows.py --flavour WIN_GPU_MKLDNN 2021-03-29 02:03:58,868 MXNet Windows build helper 2021-03-29 02:03:58,868 Build flavour: WIN_GPU_MKLDNN 2021-03-29 02:03:58,868 Detected Windows platform 2021-03-29 02:03:58,868 Using vcvars environment: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat 2021-03-29 02:03:58,868 Found MXNet root: D:\incubator-mxnet 2021-03-29 02:03:59,158 Generating project with CMake: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" && cmake -GNinja -DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=cl -DUSE_CUDA=ON -DUSE_CUDNN=ON -DENABLE_CUDA_RTC=ON -DUSE_OPENCV=ON -DUSE_OPENMP=ON -DUSE_BLAS=open -DUSE_LAPACK=ON -DUSE_DIST_KVSTORE=OFF -DMXNET_CUDA_ARCH="5.2" -DUSE_MKLDNN=ON -DCMAKE_BUILD_TYPE=Release D:\incubato r-mxnet
Visual Studio 2019 Developer Command Prompt v16.9.2 Copyright (c) 2021 Microsoft Corporation
[vcvarsall.bat] Environment initialized for: 'x64' -- The C compiler identification is MSVC 19.28.29913.0 -- The CXX compiler identification is MSVC 19.28.29913.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- CMAKE_CROSSCOMPILING FALSE -- CMAKE_HOST_SYSTEM_PROCESSOR AMD64 -- CMAKE_SYSTEM_PROCESSOR AMD64 -- CMAKE_SYSTEM_NAME Windows -- CMake version '3.20.0-rc4' using generator 'Ninja' -- Looking for a CUDA compiler -- Looking for a CUDA compiler - C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/bin/nvcc.exe -- The CUDA compiler identification is NVIDIA 10.2.89 -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/bin/nvcc.exe - skipped -- Detecting CUDA compile features -- Detecting CUDA compile features - done -- The ASM_MASM compiler identification is MSVC -- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/ml64.exe CMake Deprecation Warning at 3rdparty/mkldnn/CMakeLists.txt:17 (cmake_minimum_required): Compatibility with CMake < 2.8.12 will be removed from a future version of CMake.
Update the VERSION argument
value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. -- Intel MKL-DNN compat: set DNNL_BUILD_EXAMPLES to MKLDNN_BUILD_EXAMPLES with value
OFF
-- Intel MKL-DNN compat: set DNNL_BUILD_TESTS to MKLDNN_BUILD_TESTS with valueOFF
-- Intel MKL-DNN compat: set DNNL_ENABLE_JIT_PROFILING to MKLDNN_ENABLE_JIT_PROFILING with valueOFF
-- Intel MKL-DNN compat: set DNNL_LIBRARY_TYPE to MKLDNN_LIBRARY_TYPE with valueSTATIC
-- Intel MKL-DNN compat: set DNNL_ARCH_OPT_FLAGS to MKLDNN_ARCH_OPT_FLAGS with value `` -- Looking for pthread.h -- Looking for pthread.h - not found -- Found Threads: TRUE -- Found OpenMP_C: -openmp (found version "2.0") -- Found OpenMP_CXX: -openmp (found version "2.0") -- Found OpenMP: TRUE (found version "2.0") -- GPU support is disabled -- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.29.2.windows.3") -- Primitive cache is enabled -- Found OpenBLAS libraries: D:/OpenBLAS-v0.2.19-Win64-int32/lib/libopenblas.dll.a -- Found OpenBLAS include: D:/OpenBLAS-v0.2.19-Win64-int32/include -- OpenCV ARCH: x64 -- OpenCV RUNTIME: vc15 -- OpenCV STATIC: OFF -- Found OpenCV: C:/.conan/87a46b/1 (found version "4.1.1") found components: core highgui imgproc imgcodecs -- Found OpenCV 4.1.1 in C:/.conan/87a46b/1/cmake/x64/vc15/lib -- You might need to add C:.conan\87a46b\1\cmake\x64\vc15\bin to your PATH to be able to run your applications. -- OpenCV 4.1.1 found (C:/.conan/87a46b/1/cmake/x64/vc15/lib) -- OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs USE_LAPACK is ON CMake Warning at 3rdparty/googletest/googletest/CMakeLists.txt:47 (project): VERSION keyword not followed by a value or was followed by a value that expanded to nothing.CMake Deprecation Warning at 3rdparty/googletest/googletest/CMakeLists.txt:49 (cmake_minimum_required): Compatibility with CMake < 2.8.12 will be removed from a future version of CMake.
Update the VERSION argument
value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. -- Found PythonInterp: D:/incubator-mxnet/ci/venv/Scripts/python.exe (found version "3.6.8") -- Found GTest: gtest -- Found CUDNN: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/lib/x64/cudnn.lib -- Looking for clock_gettime in rt -- Looking for clock_gettime in rt - not found -- Looking for fopen64 -- Looking for fopen64 - not found -- Looking for C++ include cxxabi.h -- Looking for C++ include cxxabi.h - not found -- Looking for nanosleep -- Looking for nanosleep - not found -- Looking for backtrace -- Looking for backtrace - not found -- Could NOT find Backtrace (missing: Backtrace_LIBRARY Backtrace_INCLUDE_DIR) -- D:/incubator-mxnet/3rdparty/dmlc-core/cmake/build_config.h.in -> include/dmlc/build_config.h -- CUDA: Using the following NVCC architecture flags -gencode;arch=compute_52,code=sm_52 -- Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/include (found version "10.2.89") -- Performing Test SUPPORT_MSSE3 -- Performing Test SUPPORT_MSSE3 - Failed -- Performing Test SUPPORT_MSSE2 -- Performing Test SUPPORT_MSSE2 - Failed -- Determining F16C support F16C instruction set is not yet supported for MSVC -- CUDA: Adding NVCC options: --fatbin-options --compress-all -- Google Test not found -- Configuring done CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.
CUDA_ARCHITECTURES is empty for target "mxnet_52". This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt: Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC, empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104" for policy details. Use the cmake_policy command to set the policy and suppress this warning.
CUDA_ARCHITECTURES is empty for target "customop_gpu_lib". This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done -- Build files have been written to: D:/incubator-mxnet/ci/build 2021-03-29 02:04:21,538 Building: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" && ninja
Visual Studio 2019 Developer Command Prompt v16.9.2 Copyright (c) 2021 Microsoft Corporation
[vcvarsall.bat] Environment initialized for: 'x64' [2/877] Building CXX object CMakeFiles\mxnet_52.dir\src\operator\identity_attach_KL_sparse_reg.cc.obj
After 1800000 more lines build log ends with ninja: build stopped: subcommand failed. Quick search by words "error" or "failed" gives this:
[571/877] Building CUDA object CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj FAILED: CMakeFiles/mxnet_52.dir/src/operator/contrib/adaptive_avg_pooling.cu.obj C:\PROGRA~1\NVIDIA~2\CUDA\v10.2\bin\nvcc.exe -forward-unknown-to-host-compiler -DCUB_IGNORE_DEPRECATED_CPP_DIALECT -DDMLC_CORE_USE_CMAKE -DDMLC_LOG_FATAL_THROW=1 -DDMLC_LOG_STACK_TRACE_SIZE=0 -DDMLC_MODERN_THREAD_LOCAL=0 -DDMLC_STRICT_CXX11 -DDMLC_USE_CXX11 -DDMLC_USE_CXX11=1 -DMSHADOW_FORCE_STREAM -DMSHADOW_INT64_TENSOR_SIZE=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_CUDA=1 -DMSHADOW_USE_CUDNN=1 -DMSHADOW_USE_F16C=0 -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_SSE=0 -DMXNET_ENABLE_CUDA_RTC=1 -DMXNET_EXPORTS -DMXNET_USE_BLAS_OPEN=1 -DMXNET_USE_CUDA=1 -DMXNET_USE_LAPACK=1 -DMXNET_USE_MKLDNN=1 -DMXNET_USE_OPENCV=1 -DMXNET_USE_OPENMP=1 -DMXNET_USE_SIGNAL_HANDLER=1 -DNNVM_EXPORTS -DNOMINMAX -DUSE_CUDNN -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_WARNINGS -D__USE_XOPEN2K8 -Dmxnet_52_EXPORTS -I....\3rdparty\mkldnn\include -I3rdparty\mkldnn\include -I....\include -I....\src -I....\3rdparty\tvm\nnvm\include -I....\3rdparty\tvm\include -I....\3rdparty\dmlc-core\include -I....\3rdparty\dlpack\include -I....\3rdparty\nvidia_cub -I....\3rdparty\mshadow -I....\3rdparty\mkldnn\src..\include -I3rdparty\dmlc-core\include -isystem=D:\OpenBLAS-v0.2.19-Win64-int32\include -isystem=C:.conan\87a46b\1\include -isystem="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include" -D_WINDOWS -Xcompiler="/W3 /GR /EHsc" --fatbin-options --compress-all -Xcompiler="-MD -O2 -Ob2" -DNDEBUG --gpu-architecture=compute_52 --gpu-code=sm_52,compute_52 "-Xcompiler=-MT -Gy /bigobj" -MD -MT CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj -MF CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj.d -x cu -c ....\src\operator\contrib\adaptive_avg_pooling.cu -o CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj -Xcompiler=-FdCMakeFiles\mxnet_52.dir\,-FS
There's also the same error message for bilinear_sampler.cu.obj and deformable_psroi_pooling.cu.obj. Can you advise next step?
Please check the failing files in the v1.x / master branch. There might be fixes that need to be backported for v1.8.x to work
Unfortunately I'm still facing same errors. Perhaps something is wrong with CUDA installation on my machine, or with CUDA integration with Visual Studio. Are you actually cross-compiling MXNet in Docker container? I want to try to build this way now, but I don't quite understand what exactly should I do.
You could try updating to latest cuda and latest msvc. We don't cross-compile for windows
@leezu Turns out the solution was not to update to the latest msvs but to downgrade to the previous version: https://devtalk.blender.org/t/cuda-compile-error-windows-10/17886/5 When I switched from 16.9 to 16.8.6, these errors disappeared. But I still got some (in 3 build attempts in a row):
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2993: "T": illegal type for non-type template parameter "formal" D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2065: T289: undeclared identifier D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2923: std::_Select<__formal>::_Apply: "__T289" is not a valid template type argument for parameter "
" D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2062: type 'unknown-type' unexpected
Quick search led to this issue https://github.com/NVIDIA/thrust/issues/1090 but I didn't find any Thrust headers in util_type.cuh. I'm also not sure if error depends on USE_CPP_PACKAGE=ON setting, so I opened a PR just to test it: https://github.com/apache/incubator-mxnet/pull/20108 In the meantime, will try to build with CUDA 11 and report back. I honestly think that nightly builds of libmxnet.dll/.so would greatly improve MXNet accessibility.
Btw, what is the exact version of MSVS that CI uses?
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2993: "T": illegal type for non-type template parameter "__formal" D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2065: __T289: undeclared identifier D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2923: std::_Select<__formal>::_Apply: "__T289" is not a valid template type argument for parameter "" D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2062: type 'unknown-type' unexpected
This is a bug in CUDA - MSVC integration and should be solved by upgrading to Cuda 11. Note that this issue is not thrust specific, but a bug in nvcc compiler. That's why it occurs for you even without thrust. For more details, see https://github.com/thrust/thrust/issues/1090
@josephevans may know the precise version. Alternativel, I think you can look at the compilation log of the CI build which should contain a line with the version number
@leezu Yes, building with CUDA 11 did help. Maybe CUDA version in build_windows.py should reflect that? Anyway, I'm too lazy for another PR
Description
mxnet_60 project fails to build
Error Message
Lots of them but only of two kinds:
Code Description Project File Line
Rest of the errors are the same except for name of the function or file
Steps to reproduce
What have you tried to solve it?
compiler is out of heap space in pass 2
error for the same project which I couldn't resolve (as suggested somewhere here) by settings 'Link Time Code Optimization' to '/LTCG:PGOptimize' because VS can't find mxnet_60.pgd file. However, I've successfully built version without CUDA with VS 2019.Environment