search

facebookresearch / xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.
https://facebookresearch.github.io/xformers/
Other
8.67k stars 619 forks source link

Cant use powershell command to build for Nightly PyTorch on Win - filename too long #1132

Open Mescalamba opened 1 month ago

Mescalamba commented 1 month ago

Ive tried to use command to build for nightly PyTorch version on Win, problem is that it uses filenames waaay too long, so it just throws error and does obviously nothing.

Not sure if its Python 3.11.9 issue or not.

Any way around? I do have long paths enabled, but it doesnt seem to have any effect (probably cause app itself needs to be able to use long paths).

JohnnyStreet commented 1 month ago

Same issue.. I can't install it from PyPi or from the git anymore on Windows, and I have enabled long filenames both in the registry and through the group policy editor. A workaround is to change your TEMP environment variable and reboot, then install, but you will have to change it back and reboot again after. This is not a reasonable ask for end users who are trying to install my requirements.

xiezhipeng-git commented 3 weeks ago

same issue.

Emitting ninja build file C:\Users\Admin\AppData\Local\Temp\pip-install-3v32sn_o\xformers_a31d546079914685bf9f1
6b55476c94d\build\temp.win-amd64-cpython-310\Release\build.ninja...
      Compiling objects...
      Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=
N)
      ninja: error: Stat(C:/Users/Admin/AppData/Local/Temp/pip-install-3v32sn_o/xformers_a31d546079914685bf9f16b55476
c94d/build/temp.win-amd64-cpython-310/Release/Users/Admin/AppData/Local/Temp/pip-install-3v32sn_o/xformers_a31d546079
914685bf9f16b55476c94d/third_party/flash-attention/csrc/flash_attn/flash_api.obj): Filename longer than 260 character
s
      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\lib\site-packages\torch\utils\cpp_extension.py", line 2104, in _run_ninja_build
          subprocess.run(
        File "D:\my\env\python3.10.10\lib\subprocess.py", line 526, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Admin\AppData\Local\Temp\pip-install-3v32sn_o\xformers_a31d546079914685bf9f16b55476c94d\setup.
py", line 680, in <module>
          setuptools.setup(
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\core.py", line 184, in setup
          return run_commands(dist)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\core.py", line 200, in run_commands
          dist.run_commands()
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
          self.run_command(cmd)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\dist.py", line 950, in run_command
          super().run_command(command)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
          cmd_obj.run()
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\command\bdist_wheel.py", line 384, in run
          self.run_command("build")
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\dist.py", line 950, in run_command
          super().run_command(command)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
          cmd_obj.run()
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
          self.run_command(cmd_name)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\dist.py", line 950, in run_command
          super().run_command(command)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
          cmd_obj.run()
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\command\build_ext.py", line 98, in run
          _build_ext.run(self)
        File "D:\my\env\python3.10.10\lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 359, in run
          self.build_extensions()
        File "C:\Users\Admin\AppData\Local\Temp\pip-install-3v32sn_o\xformers_a31d546079914685bf9f16b55476c94d\setup.
py", line 637, in build_extensions
          super().build_extensions()
        File "D:\my\env\python3.10.10\lib\site-packages\torch\utils\cpp_extension.py", line 868, in build_extensions
          build_ext.build_extensions(self)
        File "D:\my\env\python3.10.10\lib\site-packages\Cython\Distutils\old_build_ext.py", line 195, in build_extens
ions
          _build_ext.build_ext.build_extensions(self)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 476, in bui
ld_extensions
          self._build_extensions_serial()
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 502, in _bu
ild_extensions_serial
          self.build_extension(ext)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\command\build_ext.py", line 263, in build_extensio
n
          _build_ext.build_extension(self, ext)
        File "D:\my\env\python3.10.10\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 557, in bui
ld_extension
          objects = self.compiler.compile(
        File "D:\my\env\python3.10.10\lib\site-packages\torch\utils\cpp_extension.py", line 840, in win_wrap_ninja_co
mpile
          _write_ninja_file_and_compile_objects(
        File "D:\my\env\python3.10.10\lib\site-packages\torch\utils\cpp_extension.py", line 1784, in _write_ninja_fil
e_and_compile_objects
          _run_ninja_build(
        File "D:\my\env\python3.10.10\lib\site-packages\torch\utils\cpp_extension.py", line 2120, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xformers
  Running setup.py clean for xformers
Failed to build xformers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (xformers)
IzzyHibbert commented 2 weeks ago

+1 Same issue: does not compile under windows. Tried and tried also to modify the longpath under windows and then reboot. No way, doesn't help/fix...

  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  ninja: error: Stat(C:/Users/37019Administrator/AppData/Local/Temp/pip-install-7o9vf35s/xformers_f5bdb208bf33471b8dcad9cf40706dce/build/temp.win-amd64-cpython-311/Release/Users/37019Administrator/AppData/Local/Temp/pip-install-7o9vf35s/xformers_f5bdb208bf33471b8dcad9cf40706dce/third_party/flash-attention/csrc/flash_attn/flash_api.obj): Filename longer than 260 characters
  Traceback (most recent call last):
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\torch\utils\cpp_extension.py", line 2105, in _run_ninja_build
      subprocess.run(
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "C:\Users\37019Administrator\AppData\Local\Temp\pip-install-7o9vf35s\xformers_f5bdb208bf33471b8dcad9cf40706dce\setup.py", line 680, in <module>
      setuptools.setup(
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
      dist.run_commands()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
      self.run_command(cmd)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\dist.py", line 950, in run_command
      super().run_command(command)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
      cmd_obj.run()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\command\bdist_wheel.py", line 398, in run
      self.run_command("build")
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\dist.py", line 950, in run_command
      super().run_command(command)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
      cmd_obj.run()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\dist.py", line 950, in run_command
      super().run_command(command)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
      cmd_obj.run()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\command\build_ext.py", line 98, in run
      _build_ext.run(self)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 359, in run
      self.build_extensions()
    File "C:\Users\37019Administrator\AppData\Local\Temp\pip-install-7o9vf35s\xformers_f5bdb208bf33471b8dcad9cf40706dce\setup.py", line 637, in build_extensions
      super().build_extensions()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\torch\utils\cpp_extension.py", line 866, in build_extensions
      build_ext.build_extensions(self)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 476, in build_extensions
      self._build_extensions_serial()
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 502, in _build_extensions_serial
      self.build_extension(ext)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\command\build_ext.py", line 263, in build_extension
      _build_ext.build_extension(self, ext)
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 557, in build_extension
      objects = self.compiler.compile(
                ^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\torch\utils\cpp_extension.py", line 838, in win_wrap_ninja_compile
      _write_ninja_file_and_compile_objects(
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\torch\utils\cpp_extension.py", line 1785, in _write_ninja_file_and_compile_objects
      _run_ninja_build(
    File "C:\Users\37019Administrator\anaconda3\envs\unsloth_env\Lib\site-packages\torch\utils\cpp_extension.py", line 2121, in _run_ninja_build
      raise RuntimeError(message) from e
  RuntimeError: Error compiling objects for extension
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for xformers Running setup.py clean for xformers Failed to build xformers

lw commented 2 weeks ago

We don't have expertise on Windows so we're not sure how we can help.

Our CI is regularly building on Windows so you could check there to see how we're doing it.

You could try passing --no-build-isolation to pip when building, this may avoid it copying the source to a temporary directory.

WXizhi commented 2 weeks ago

Firstly, I cloned the repository, instead of automatically downloading to the cache directory. Then remove the win10 filename constraint by: "win+R -> gpedit.msc -> Computer Configuration -> Administrative Templates -> System -> FileSystem -> Enable win32 long paths".

I have another error: catastrophic error, out of memory. The solution is to reduce the number of ninja workers: set MAX_JOBS=4 to avoid high memory utility.

It works in my case on Win10 + torch2.5.1 + cu121 + VS2022 x64