search

SHI-Labs / NATTEN

Neighborhood Attention Extension. Bringing attention to a neighborhood near you!
https://shi-labs.com/natten/
Other
363 stars 29 forks source link

install error from source? #134

Open TongkunGuan opened 5 months ago

TongkunGuan commented 5 months ago

`guantongkun in 🌐 DVLab-1 in NATTEN on  main via 🐍 v3.8.19 via 🅒 BA-SAM took 2s ❯ make Cleaning up rm -rf dist/ rm -rf natten.egg-info/ rm -rf src/natten/_C. rm -rf src/natten/libnatten. rm -rf pycache rm -rf tests/pycache rm -rf src/pycache rm -rf src/natten/pycache rm -rf src/natten.egg rm -rf install.out Uninstalling NATTEN pip uninstall -y natten WARNING: Skipping natten as it is not installed. Fetching all third party submodules git submodule update --init --recursive Installing NATTEN from source NATTEN_CUDA_ARCH="" \ NATTEN_N_WORKERS="" \ NATTEN_WITH_CUDA="" \ NATTEN_VERBOSE="" \ pip install -v -e . 2>&1 | tee install.out Using pip 24.0 from /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/pip (python 3.8) Obtaining file:///home/guantongkun/TS/NATTEN Preparing metadata (setup.py): started Running command python setup.py egg_info Building NATTEN with CUDA 118 Building NATTEN for SM: 8.6 Number of workers: 18 running egg_info creating /tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info writing /tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/dependency_links.txt writing requirements to /tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/requires.txt writing top-level names to /tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.cc' under directory 'csrc' warning: no files found matching '.cpp' under directory 'third_party/cutlass/include' warning: no files found matching '.cuh' under directory 'third_party/cutlass/include' warning: no files found matching '.cu' under directory 'third_party/cutlass/include' warning: no files found matching '.txt' under directory 'third_party/cutlass/include' warning: no files found matching '*.cc' under directory 'third_party/cutlass/include' no previously-included directories found matching 'webpage/' no previously-included directories found matching 'tools/' no previously-included directories found matching 'scripts/' adding license file 'LICENSE' writing manifest file '/tmp/pip-pip-egg-info-rkgk5ssc/natten.egg-info/SOURCES.txt' Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: packaging in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from natten==0.17.1) (24.0) Requirement already satisfied: torch>=2.0.0 in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from natten==0.17.1) (2.0.0+cu118) Requirement already satisfied: filelock in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (3.13.1) Requirement already satisfied: typing-extensions in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (4.9.0) Requirement already satisfied: sympy in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (1.12) Requirement already satisfied: networkx in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (3.0) Requirement already satisfied: jinja2 in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (3.1.3) Requirement already satisfied: triton==2.0.0 in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from torch>=2.0.0->natten==0.17.1) (2.0.0) Requirement already satisfied: cmake in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from triton==2.0.0->torch>=2.0.0->natten==0.17.1) (3.25.0) Requirement already satisfied: lit in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from triton==2.0.0->torch>=2.0.0->natten==0.17.1) (15.0.7) Requirement already satisfied: MarkupSafe>=2.0 in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from jinja2->torch>=2.0.0->natten==0.17.1) (2.1.5) Requirement already satisfied: mpmath>=0.19 in /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages (from sympy->torch>=2.0.0->natten==0.17.1) (1.3.0) Installing collected packages: natten Running setup.py develop for natten Running command python setup.py develop Building NATTEN with CUDA 118 Building NATTEN for SM: 8.6 Number of workers: 18 running develop running egg_info creating src/natten.egg-info writing src/natten.egg-info/PKG-INFO writing dependency_links to src/natten.egg-info/dependency_links.txt writing requirements to src/natten.egg-info/requires.txt writing top-level names to src/natten.egg-info/top_level.txt writing manifest file 'src/natten.egg-info/SOURCES.txt' reading manifest file 'src/natten.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
warning: no files found matching '*.cc' under directory 'csrc'
warning: no files found matching '*.cpp' under directory 'third_party/cutlass/include'
warning: no files found matching '*.cuh' under directory 'third_party/cutlass/include'
warning: no files found matching '*.cu' under directory 'third_party/cutlass/include'
warning: no files found matching '*.txt' under directory 'third_party/cutlass/include'
warning: no files found matching '*.cc' under directory 'third_party/cutlass/include'
no previously-included directories found matching 'webpage/'
no previously-included directories found matching 'tools/'
no previously-included directories found matching 'scripts/'
adding license file 'LICENSE'
writing manifest file 'src/natten.egg-info/SOURCES.txt'
running build_ext
-- The CXX compiler identification is GNU 11.4.0
-- The CUDA compiler identification is unknown
CMake Error at /home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeDetermineCUDACompiler.cmake:603 (message):
  Failed to detect a default CUDA architecture.

  Compiler output:

Call Stack (most recent call first):
  CMakeLists.txt:5 (project)

-- Configuring incomplete, errors occurred!
See also "/home/guantongkun/TS/NATTEN/build/lib.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log".
See also "/home/guantongkun/TS/NATTEN/build/lib.linux-x86_64-cpython-38/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "<pip-setuptools-caller>", line 34, in <module>
  File "/home/guantongkun/TS/NATTEN/setup.py", line 243, in <module>
    setup(
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/__init__.py", line 104, in setup
    return distutils.core.setup(**attrs)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
    return run_commands(dist)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
    dist.run_commands()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
    super().run_command(command)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/command/develop.py", line 111, in install_for_development
    self.run_command('build_ext')
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
    super().run_command(command)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 91, in run
    _build_ext.run(self)
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
    self.build_extensions()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
    self._build_extensions_serial()
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/guantongkun/TS/NATTEN/setup.py", line 219, in build_extension
    subprocess.check_call(
  File "/home/guantongkun/.conda/envs/BA-SAM/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/guantongkun/TS/NATTEN/csrc', '-DPYTHON_PATH=/home/guantongkun/.conda/envs/BA-SAM/bin/python', '-DOUTPUT_FILE_NAME=natten/libnatten.cpython-38-x86_64-linux-gnu', '-DNATTEN_CUDA_ARCH_LIST=86-real', '-DNATTEN_IS_WINDOWS=0', '-DNATTEN_IS_MAC=0', '-DIS_LIBTORCH_BUILT_WITH_CXX11_ABI=0', '-DNATTEN_WITH_AVX=1', '-DNATTEN_WITH_CUDA=1', '-DNATTEN_WITH_CUTLASS=1']' returned non-zero exit status 1.
Current arch list: [86] (max: 86)
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/guantongkun/.conda/envs/BA-SAM/bin/python -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
#   import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
#   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
#     manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize

try:
    import setuptools
except ImportError as error:
    print(
        "ERROR: Can not execute `setup.py` since setuptools is not available in "
        "the build environment.",
        file=sys.stderr,
    )
    sys.exit(1)

__file__ = %r
sys.argv[0] = __file__

if os.path.exists(__file__):
    filename = __file__
    with tokenize.open(__file__) as f:
        setup_py_code = f.read()
else:
    filename = "<auto-generated setuptools caller>"
    setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/home/guantongkun/TS/NATTEN/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
cwd: /home/guantongkun/TS/NATTEN/

error: subprocess-exited-with-error

× python setup.py develop did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. `

alihassanijr commented 5 months ago

It looks like cmake is failing to detect your CUDA compiler.

Is there a reason you're building from source? I don't think there's been any changes since the last release, 0.17.1.

TongkunGuan commented 5 months ago

It looks like cmake is failing to detect your CUDA compiler.

Is there a reason you're building from source? I don't think there's been any changes since the last release, 0.17.1.

Because I want to use kernel=65. When using pip3 install natten==0.17.1+torch200cu118 -f https://shi-labs.com/natten/wheels, the following issue will appear:

NATTEN kernel launch failed! na2d_pn_cuda_gemm_sm80_float does not support implement kernel size 65. You may try generating it manually and build from source. Refer to NATTEN's github repository for more information.

alihassanijr commented 5 months ago

Yeah so you're trying to use the unfused interface; unless you need to train with positional bias, I highly recommend using the FNA interface. I.e. instead of importing na2d_qk and na2d_av from natten.functional, just import na2d. Read more here:

https://github.com/SHI-Labs/NATTEN/blob/main/docs/fna/fused-vs-unfused.md

If you need to use unfused, you should be able to just run naive kernels with any kernel size by disabling gemm:

from natten import disable_gemm_na

disable_gemm_na()

If you want to use the unfused GEMM kernels (i.e. need performance and need to use unfused because of RPB), please let me know, you'll have to re-generate the GEMM instantiations, and then build from source.

TongkunGuan commented 5 months ago

Yeah so you're trying to use the unfused interface; unless you need to train with positional bias, I highly recommend using the FNA interface. I.e. instead of importing na2d_qk and na2d_av from natten.functional, just import na2d. Read more here:

https://github.com/SHI-Labs/NATTEN/blob/main/docs/fna/fused-vs-unfused.md

If you need to use unfused, you should be able to just run naive kernels with any kernel size by disabling gemm:

from natten import disable_gemm_na

disable_gemm_na()

If you want to use the unfused GEMM kernels (i.e. need performance and need to use unfused because of RPB), please let me know, you'll have to re-generate the GEMM instantiations, and then build from source.

Thanks! Some backbones (MViT, SAM) use the rel_pos_bias different from the rpb. How should I solve this problem?

SAM

alihassanijr commented 5 months ago

Yeah so as long as you're on the latest NATTEN release (0.17.1), and run disable_gemm_na() before the first call to NATTEN (i.e. the beginning of your program), kernel size 65 should work, but it will fall back to our naive kernels, which might not be too performant. If you'd like you use our GEMM kernels, you would need to build from source, but first, you'll have to remove the generated GEMM kernels and re-generate with max kernel size set to a number larger than 65:

rm -rf csrc/autogen/include/natten_autogen/cuda/gemm/2d
rm -rf csrc/autogen/src/cuda/gemm/2d

python3 scripts/autogen_cuda_gemm_2d.py --max-kernel-size 67

And then build from source. But from the looks of it, you'll need to figure out the cmake issue first. I'd recommend trying installing the cuda toolkit with anaconda or use a docker image.

Thanks! Some backbones (MViT, SAM) use the rel_pos_bias different from the rpb. How should I solve this problem?

I'm not sure I understand; if it's a relative positional bias, it should be dependent on the attention pattern, not model, so you'd use the RPB in NATTEN. However, I wouldn't use RPB or any explicit biasing of attention weights. There's better alternatives that are more easily computable.

TongkunGuan commented 5 months ago

I'm not sure I understand; if it's a relative positional bias, it should be dependent on the attention pattern, not model, so you'd use the RPB in NATTEN.

SAM SAM SAM

alihassanijr commented 23 hours ago

It's been a while since I looked at this issue. Are you still having issues?