Inability to install pytorch3d due to torch not found

leweex95 commented 11 months ago

🐛 Bugs / Unexpected behaviors

Versions

Python: 3.8.10 OS: Ubuntu 20.04.5 LTS Poetry: 1.6.1 pip: 23.2.1

Description

I am trying to set up a poetry environment with this library installed. I started off with a completely new, empty environment. Then I attempted to add the poetry3d package:

poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git

Instead of adding it to the poetry environment, I got the following error:

Cloning https://github.com/facebookresearch/pytorch3d.git at 'HEAD' to /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d

  Stack trace:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:344 in _run
      342│         try:
      343│             if input_:
    → 344│                 output: str = subprocess.run(
      345│                     cmd,
      346│                     stdout=subprocess.PIPE,

  CalledProcessError

  Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', '-']' returned non-zero exit status 1.

  at /usr/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:

  Stack trace:

  3  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:601 in get_pep517_metadata
      599│                 *PEP517_META_BUILD_DEPS,
      600│             )
    → 601│             venv.run(
      602│                 "python",
      603│                 "-",

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
      312│     def run(self, bin: str, *args: str, **kwargs: Any) -> str:
      313│         cmd = self.get_command_from_bin(bin) + list(args)
    → 314│         return self._run(cmd, **kwargs)
      315│ 
      316│     def run_pip(self, *args: str, **kwargs: Any) -> str:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
       94│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       95│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    →  96│         return super()._run(cmd, **kwargs)
       97│ 
       98│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1

  Output:
  Traceback (most recent call last):
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
      yield
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
      self.gen.throw(type, value, traceback)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
      raise BuildBackendException(  # noqa: B904 # use raise from
  build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel

  Input:
  import build
  import build.env
  import pyproject_hooks

  source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
  dest = '/tmp/tmpxdbpwgq0/dist'

  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pyproject_hooks.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
      359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )
      362│         except CalledProcessError as e:
    → 363│             raise EnvCommandError(e, input=input_)
      364│ 
      365│         return output
      366│ 
      367│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  Stack trace:

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:359 in _run
      357│                 output = ""
      358│             else:
    → 359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )

  1  /usr/lib/python3.8/subprocess.py:415 in check_output
       413│         kwargs['input'] = empty
       414│ 
    →  415│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       416│                **kwargs).stdout
       417│ 

  CalledProcessError

  Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info']' returned non-zero exit status 1.

  at /usr/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:

  Stack trace:

  3  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:622 in get_pep517_metadata
      620│             os.chdir(path)
      621│             try:
    → 622│                 venv.run("python", "setup.py", "egg_info")
      623│                 info = PackageInfo.from_metadata(path)
      624│             except EnvCommandError as fbe:

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
      312│     def run(self, bin: str, *args: str, **kwargs: Any) -> str:
      313│         cmd = self.get_command_from_bin(bin) + list(args)
    → 314│         return self._run(cmd, **kwargs)
      315│ 
      316│     def run_pip(self, *args: str, **kwargs: Any) -> str:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
       94│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       95│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    →  96│         return super()._run(cmd, **kwargs)
       97│ 
       98│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1

  Output:
  Traceback (most recent call last):
    File "setup.py", line 15, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
      359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )
      362│         except CalledProcessError as e:
    → 363│             raise EnvCommandError(e, input=input_)
      364│ 
      365│         return output
      366│ 
      367│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  Stack trace:

  18  ~/.local/lib/python3.8/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  17  ~/.local/lib/python3.8/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  16  ~/.local/lib/python3.8/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  15  ~/.local/lib/python3.8/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  14  ~/.local/lib/python3.8/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  13  ~/.local/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  12  ~/.local/lib/python3.8/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

  11  ~/.local/lib/python3.8/site-packages/poetry/console/commands/add.py:160 in handle
       158│             return 0
       159│ 
     → 160│         requirements = self._determine_requirements(
       161│             packages,
       162│             allow_prereleases=self.option("allow-prereleases"),

  10  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:375 in _determine_requirements
       373│ 
       374│         result = []
     → 375│         for requirement in self._parse_requirements(requires):
       376│             if "git" in requirement or "url" in requirement or "path" in requirement:
       377│                 result.append(requirement)

   9  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in _parse_requirements
       439│             cwd=cwd,
       440│         )
     → 441│         return [parser.parse(requirement) for requirement in requirements]
       442│ 
       443│     def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:

   8  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in <listcomp>
       439│             cwd=cwd,
       440│         )
     → 441│         return [parser.parse(requirement) for requirement in requirements]
       442│ 
       443│     def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:

   7  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:89 in parse
        87│ 
        88│         specification = (
     →  89│             self._parse_url(requirement)
        90│             or self._parse_path(requirement)
        91│             or self._parse_simple(requirement)

   6  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:149 in _parse_url
       147│ 
       148│         if url_parsed.scheme in GIT_URL_SCHEMES:
     → 149│             return self._parse_git_url(requirement)
       150│ 
       151│         if url_parsed.scheme in ["http", "https"]:

   5  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:133 in _parse_git_url
       131│ 
       132│         source_root = self._env.path.joinpath("src") if self._env else None
     → 133│         package = self._direct_origin.get_package_from_vcs(
       134│             "git",
       135│             url=url.url,

   4  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:111 in get_package_from_vcs
       109│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       110│ 
     → 111│         return _get_package_from_git(
       112│             url=url,
       113│             branch=branch,

   3  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:46 in _get_package_from_git
        44│         path = path.joinpath(subdirectory)
        45│ 
     →  46│     package = DirectOrigin.get_package_from_directory(path)
        47│     package._source_type = "git"
        48│     package._source_url = url

   2  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:75 in get_package_from_directory
        73│     @classmethod
        74│     def get_package_from_directory(cls, directory: Path) -> Package:
     →  75│         return PackageInfo.from_directory(path=directory).to_package(root_dir=directory)
        76│ 
        77│     def get_package_from_url(self, url: str) -> Package:

   1  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:499 in from_directory
       497│                         info = cls.from_setup_files(path)
       498│                     else:
     → 499│                         info = get_pep517_metadata(path)
       500│                 except PackageInfoError:
       501│                     if not info:

  PackageInfoError

  Unable to determine package info for path: /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1

  Output:
  Traceback (most recent call last):
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
      yield
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
      self.gen.throw(type, value, traceback)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
      raise BuildBackendException(  # noqa: B904 # use raise from
  build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel

  Input:
  import build
  import build.env
  import pyproject_hooks

  source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
  dest = '/tmp/tmpxdbpwgq0/dist'

  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pyproject_hooks.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)

  Fallback egg_info generation failed.

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1

  Output:
  Traceback (most recent call last):
    File "setup.py", line 15, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'

  at ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:625 in get_pep517_metadata
      621│             try:
      622│                 venv.run("python", "setup.py", "egg_info")
      623│                 info = PackageInfo.from_metadata(path)
      624│             except EnvCommandError as fbe:
    → 625│                 raise PackageInfoError(
      626│                     path, e, "Fallback egg_info generation failed.", fbe
      627│                 )
      628│             finally:
      629│                 os.chdir(cwd)

Alright, so pytorch3d has a dependency on torch but despite this pre-requisite, the installation for some reason doesn't install the dependency. So I went ahead and installed torch:

poetry add torch

Which went successfully. Now I retried the previous installation command, knowing that torch is certainly included in my environment by now:

poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git

But the error is exactly the same.

Edit 1

I saw this issue from earlier: https://github.com/facebookresearch/pytorch3d/pull/1619. But it doesn't offer a clear workaround on how to install pytorch3d with poetry. As I mentioned above, I also installed torch separately before attempting to install pytorch3d, but the installation still fails, as pytorch3d somehow doesn't see the already present torch dependency in the local venv.

Edit 2

I also came across this: https://github.com/facebookresearch/pytorch3d/pull/1479 . I assume this change would solve the issue, but it seems to have been never merged for some reason.

leweex95 commented 11 months ago

Meanwhile, the pip install works fine:

pip install git+https://github.com/facebookresearch/pytorch3d.git

pip install git+https://github.com/facebookresearch/pytorch3d.git
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/facebookresearch/pytorch3d.git
  Cloning https://github.com/facebookresearch/pytorch3d.git to /tmp/pip-req-build-4m0519uo
  Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/pytorch3d.git /tmp/pip-req-build-4m0519uo
  Resolved https://github.com/facebookresearch/pytorch3d.git to commit a8c70161a1c99c1878ae7cf312cf7907a84f01b0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: fvcore in /home/myfolder/.local/lib/python3.8/site-packages (from pytorch3d==0.7.4) (0.1.5.post20221221)
Requirement already satisfied: iopath in /home/myfolder/.local/lib/python3.8/site-packages (from pytorch3d==0.7.4) (0.1.10)
Requirement already satisfied: Pillow in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (10.1.0)
Requirement already satisfied: numpy in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (1.24.4)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from fvcore->pytorch3d==0.7.4) (5.3.1)
Requirement already satisfied: tabulate in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (0.9.0)
Requirement already satisfied: termcolor>=1.1 in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (2.3.0)
Requirement already satisfied: tqdm in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (4.66.1)
Requirement already satisfied: yacs>=0.1.6 in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (0.1.8)
Requirement already satisfied: portalocker in /home/myfolder/.local/lib/python3.8/site-packages (from iopath->pytorch3d==0.7.4) (2.8.2)
Requirement already satisfied: typing-extensions in /home/myfolder/.local/lib/python3.8/site-packages (from iopath->pytorch3d==0.7.4) (4.8.0)
Building wheels for collected packages: pytorch3d
  Building wheel for pytorch3d (setup.py) ... done
  Created wheel for pytorch3d: filename=pytorch3d-0.7.4-cp38-cp38-linux_x86_64.whl size=63584603 sha256=fd2a5cc4e9308aadb7d0b10626d1fbad5d766f29c06afb2fb6170a9301ab03fa
  Stored in directory: /tmp/pip-ephem-wheel-cache-cad_9osc/wheels/5e/a7/7c/814c3888eaaecaaf0f533faeb3c6812d82274484be51f7441c
Successfully built pytorch3d
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: pytorch3d
Successfully installed pytorch3d-0.7.4

The problem is, I want to use poetry, not pip, and in that case, I am running into the aforementioned ModuleNotFound error

bottler commented 11 months ago

I think poetry might be deliberately not (yet) designed to deal with packages like PyTorch3D which have tight build time dependencies on versions of other packages. So it might be "by design" that this doesn't work easily. If that's not the case, we'd like help, e.g. pointers to documentation. If it is possible to adapt PyTorch3D nicely into poetry, it hasn't been done.

leweex95 commented 11 months ago

Thank you for your reply @bottler .

It seems that this issue is essentially a duplicate of https://github.com/facebookresearch/pytorch3d/issues/1419, as torch is a build-time dependency of the library, but it is not declared in a pyproject.toml file, hence poetry has no way to include it in build time. As such, it seems to be not possible to install this package with poetry :/

facebookresearch / pytorch3d