cant build it from source for Windows

[neurosynapse]

Hello,

I have been trying to install natten on Windows from source. Unfortunately unsuccessful. I have tried lot of different approaches. It would be nice if you could help me in this. Here is the error I m getting.

Best regards, Roberts

[alihassanijr]

You are right, the error is in line 196, and line 196 is the line starting a subprocess that executes make.

I missed the error I was looking for, which is what you shared; translates into make: *** No target specified and makefile not found. stop.

Could you try and go into the build/ directory again, and see if cmake even generated a makefile? Just look in the directory and see whether there's a file named Makefile (no extension).

[zxl1203]

It seems that there is no file named Makefile in the build/ directory. Only the Makefile is in the NATTEN/ directory, but it originally exists. The three pictures show my file directory.

[alihassanijr]

I think this is all because cmake works differently on windows despite my original thinking. Could you try going into build and running:

cmake --build .

It should be either build or build/lib.win.amd64-cpython-38.

I greatly appreciate your help -- we should be able to get NATTEN building on Windows again soon. And your specific setup is perfect because it looks like you have an Ampere card, which should see significant speedup over previous versions.

[zxl1203]

Yeah, my GPU card is RTX3060, cuda version is 11.8, torch version is 2.0.0 . I think these conditions are enough for building NATTEN.

Following your advice，I have run cmake --build . in the directorybuild/lib.win.amd64-cpython-38. But it seems that there is still some problems. Here is the log. log.txt

It's my pleasure that my problem could help you. I'm looking forward to your NATTEN building on Windows.

Best wishes.

[alihassanijr]

Thank you!

So the error you shared indicates that cutlass was probably not cloned. Can you run this command from the main directory, and then try exactly what you tried just now?

From NATTEN/:

git submodule update --init --recursive

and then from build/lib.win.amd64-cpython-38:

cmake --build .

[RayTan183]

Hello @alihassanijr , Thank you for your patient answer. I have met the same question as @zxl1203. Following your advice metioned above, there are still some problems. When I initially follow the commond "Build from source", it can not run the commond make. And there is not build in NATTEN. The results is as follow.

Then I try the commond /pip3 install -e . > natten-out.txt 2>&1 And the build appear. Here is the folder.

After that, i also follow your commond From NATTEN/: git submodule update --init --recursive

and then from build/lib.win.amd64-cpython-38: cmake --build .

But it also seems some problems.

My cuda is 11.7 and pytorch version is 1.13. Best regards

[alihassanijr]

Thank you, could you try this?

pip3 install ninja

cmake -GNinja --build . 

And yes, GNU make is not installed on Windows, that's why you needed to run git submodule update --init --recursive and pip install -e . yourself (calling make essentially runs those with a few more other commands).

And yes, please run cmake -GNinja --build . from NATTEN/build/lib.win-amd4-3.8, not from NATTEN/.

[RayTan183]

@alihassanijr Thank you for your reply. I am so sorry that it can not work. The photo shows my problems. Best regards

[alihassanijr]

Could you try this?

cmake --build . -GNinja

[RayTan183]

It doesn't work.

[alihassanijr]

I think at this point I'd suggest using WSL. I'm really not sure how to set up cmake on Windows; I'm sorry. With WSL you should be able to just follow the original instructions and build.

[RayTan183]

By following the original instructions which is Build from source, it seems some different errors. I use the commond as follow: pip install ninja # Recommended, not required git clone https://github.com/SHI-Labs/NATTEN cd NATTEN pip install -e .

Here are the errors:

Does it means i have to set the WORKERS when building?

[alihassanijr]

This still looks like you're using the windows command line, and not WSL. You can't follow the README instructions unless you're either on a Unix-like OS (i.e. Linux and MacOS), or on Windows and using WSL.

[RayTan183]

OK. Than you for your patient reply again! Hvae a nice day!

[Daniel63656]

Ist there any Chance this will be running on windows soon?

[alihassanijr]

@Daniel63656 yes, but unfortunately the issue has always been that I don't have a Windows machine myself. What we've done in the past was asking Windows users that are willing to help to try building NATTEN on Windows and sharing their logs so we can debug the build process together. We succeeded a few months ago when we were still using the default builder, but since we've switched to CMake, it's become complicated again.

If you are willing to help us with that (which we would greatly appreciate), please let us know and we'll update our temporary windows branch and give you instructions on how to build and grab the logs.

[Daniel63656]

Yes I can do that. My pipeline is windows based and I would love to avoid switching between WSL and windows kernels al the time if possible. Should we communicate on another channel then?

[alihassanijr]

Thank you @Daniel63656 . Yes that sounds good; could you send me an email at alih@uoregon.edu ?

[ZWBSJSW]

Hello,

I have been trying to install natten on Windows from window-cmake(last week updated), but unfortunately unsuccessful. I have tried lot of different approaches. It would be nice if you could help me in this. Here is error I'm getting in the install.txt. Thank you!

Best regards, zwbsjsw install.txt

[Daniel63656]

Based on my own attempts:

You need to use the MSVC compiler to build the project (This solved the cl not found errors). Here are the steps I did to overcome this:

-download microsoft build tools for c++ from https://visualstudio.microsoft.com/de/visual-cpp-build-tools/

-select and install the dektop development with c++ option

-next you need to set the paths to the compilers as environment variables. MSVC installs the compilers in two different directories. Find paths for c++ compiler cl.exe and windows specific compiler rc.exe. For me the paths were: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.37.32822\bin\Hostx64\x64 C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64 Add these to your PATH environment variable. Make sure you don't have any other toolchain (like minGW) variable entries above your newly created ones. You can verify you set up the paths correctly by opening a terminal and typing "cl" and "rc" which should now be recognized

-next we need to create environment variables to link to native libraries. For that locate "kernel32.lib", "MSVCRTD.lib" and "ucrtd.lib". My paths were: C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22621.0\um\x64 C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.37.32822\lib\x64\uwp C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22621.0\ucrt\x64 Add these paths to your LIB variable

-follow the build from sources instructions to clone repo $ git clone https://github.com/SHI-Labs/NATTEN $ cd NATTEN

-then in your prefered environment run $ pip install -r requirements.txt $ pip install -e .

All compiler and linking errors are fixed with this and the current error is:

        -- Build files have been written to: C:/Users/Daniel/NATTEN/build/lib.win-amd64-cpython-311
        Makefile:32: *** missing separator.  Stop.
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "C:\Users\Daniel\NATTEN\setup.py", line 213, in <module>
            setup(
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\__init__.py", line 103, in setup
            return distutils.core.setup(**attrs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
            return run_commands(dist)
                   ^^^^^^^^^^^^^^^^^^
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
            dist.run_commands()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
            self.run_command(cmd)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\dist.py", line 989, in run_command
            super().run_command(command)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
            cmd_obj.run()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\command\develop.py", line 34, in run
            self.install_for_development()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\command\develop.py", line 109, in install_for_development
            self.run_command('build_ext')
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
            self.distribution.run_command(command)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\dist.py", line 989, in run_command
            super().run_command(command)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
            cmd_obj.run()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\command\build_ext.py", line 88, in run
            _build_ext.run(self)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
            self.build_extensions()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 467, in build_extensions
            self._build_extensions_serial()
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 493, in _build_extensions_serial
            self.build_extension(ext)
          File "C:\Users\Daniel\NATTEN\setup.py", line 210, in build_extension
            subprocess.check_call(['make', f"-j{n_workers}"], cwd=self.build_lib)
          File "C:\Users\Daniel\anaconda3\envs\natten\Lib\subprocess.py", line 413, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['make', '-j4']' returned non-zero exit status 2.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [106 lines of output]

I also noticed a lot of TODOs in the setup.py and I suspect it needs to be worked on to work on windows. Maybe someone wants to take it from here ;)

[ppbevilacqua]

Hello, I'm facing the same error, Is there some news or alternative to using Natten on Windows? I would use a torch model that uses this dependency. Thank you!

Best regards Paolo Pio

[alihassanijr]

I'm sorry, but my hands are tied in this matter. To my knowledge, building CUDA libraries on Windows is only possible through MSVC, but it needs a lot of debugging, and I don't happen to own or have access to a Windows machine, especially one with CUDA.

It is an issue I fully intend to fix, but it's very difficult to debug through a back and fourth with NATTEN users instead of doing it directly.

I might try to get a Windows machine with CUDA in the future, but I can't really make any commitments right now.

[earthy-zinc]

Hello, I use cmake gui to build this project. But I got stuck at this point.

<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead

Caffe2: CUDA detected: 12.1
Caffe2: CUDA nvcc is: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/bin/nvcc.exe
Caffe2: CUDA toolkit directory: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1
Caffe2: Header version is: 12.1
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib/x64/nvrtc.lib shorthash is bac8224f
USE_CUDNN is set to 0. Compiling without cuDNN support
USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
Autodetected CUDA architecture(s):  8.9
Added CUDA NVCC flags for: -gencode;arch=compute_89,code=sm_89
Found Torch: C:/Users/earthyzinc/Programs/miniconda/Lib/site-packages/torch/lib/torch.lib  
CMAKE_CUDA_FLAGS_RELEASE: -Xcompiler="-O2 -Ob2" -DNDEBUG -Xcompiler /MD -Xcompiler -O3 --use_fast_math
CMake Error at CMakeLists.txt:173 (set_target_properties):
  set_target_properties called with incorrect number of arguments.

Configuring incomplete, errors occurred!

[alihassanijr]

@earthy-zinc The issue is that you haven't specified the environment variables that the cmake file expects; specifically the target architectures.

Can you try setting them manually and trying again?

Based on what you've shared, you can do:

OUTPUT_FILE_NAME="libnatten.dll"
NATTEN_CUDA_ARCH_LIST="8.9"
NATTEN_IS_WINDOWS=1

[aneeka657]

i am also facing same problem. I cannot install natten on 64 bit windows.

i have run command : pip install -e . > natten-out.txt 2>&1 and uploaded natten-out.txt file below.

natten-out.txt

Here are my libraries installed versions.

python -c "import torch; print(torch.version); print(torch.config.show())" 2.0.1 PyTorch built with:

• C++ Version: 199711
• MSVC 193431937
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX2
• CUDA Runtime 11.7
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.5
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

please tell me how may I solve this issue?

Thanks

[alihassanijr]

@aneeka657 Could you please try this branch instead?

https://github.com/alihassanijr/NATTEN-Torch/tree/windows-cmake-2

git clone https://github.com/alihassanijr/NATTEN-Torch

cd NATTEN-Torch

git checkout windows-cmake-2

[alihassanijr]

Okay, just an FYI: NATTEN can be built in WSL without a problem. WSL support might take me a bit longer though.

[alihassanijr]

I'm marking this with wontfix, because after hours of trying to build NATTEN with MSVC, I still haven't gotten anywhere. Aside from the thousands of warnings that the compiler throws, and all the manual changes required to make to let it even get to the point of compilation, I hit linker errors.

Given how difficult this has proven to be, and the fact that NATTEN can at least work through WSL, this won't be worked on until we have someone who's had more experience with Windows than I on the project (or we end up having a lot of free time which I don't see happening soon.) Of course, contributions are always welcome.

[alihassanijr]

Fixed by #118 , but CUDA 12.X users might run into an issue with PyTorch due to https://github.com/pytorch/pytorch/pull/116926.

Please refer to the docs for instructions on avoiding this until the PyTorch PR lands and makes it to the next release.