search

entity-neural-network / incubator

Collection of in-progress libraries for entity neural networks.
Apache License 2.0
29 stars 10 forks source link

Issues setting up working environment #96

Closed dtch1997 closed 2 years ago

dtch1997 commented 2 years ago

I'm Python 3.8.12 which should be supported according to the pyproject.toml. I'm also using Poetry 1.1.7 with poetry-core==1.0.4, because of this related issue" https://stackoverflow.com/questions/69836936/poetry-attributeerror-link-object-has-no-attribute-name

However, I get weird errors concerning the concurrent installation of torch and torch-scatter. It looks like torch-scatter is trying to be installed before torch. See log.txt for full stack trace. Tried installing torch first and then re-installing torch-scatter but got a different error: log3.txt

Help appreciated

vwxyzjn commented 2 years ago

Hey @dtch1997 this is a known issue. Simply run poetry install again abs it should fix the issue. See https://github.com/entity-neural-network/incubator/blob/main/CONTRIBUTING.md#dev-setup

The error was caused by torch-scatter didn’t specify torch as a dependency.

dtch1997 commented 2 years ago

hey @vwxyzjn thanks for the response. It didn't work for me, after running the second poetry install I get this error: 

Package operations: 24 installs, 1 update, 0 removals

  • Installing torch-scatter (2.0.9): Installing...
  • Installing torch-scatter (2.0.9): Failed

  CalledProcessError

  Command '['/home/dtch1997/anaconda3/envs/entity-incubator/bin/python', '/home/dtch1997/anaconda3/envs/entity-incubator/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/dtch1997/anaconda3/envs/entity-incubator', '--no-deps', '/home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz']' returned non-zero exit status 1.

  at ~/anaconda3/envs/entity-incubator/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│
       520│

The following error occurred when trying to handle this error:

  EnvCommandError

  Command ['/home/dtch1997/anaconda3/envs/entity-incubator/bin/python', '/home/dtch1997/anaconda3/envs/entity-incubator/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/dtch1997/anaconda3/envs/entity-incubator', '--no-deps', '/home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output:
  Processing /home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/dtch1997/anaconda3/envs/entity-incubator/bin/python /tmp/tmpj_84ryar get_requires_for_build_wheel /tmp/tmpeaj8eltm
         cwd: /tmp/pip-req-build-9s_c02qi
    Complete output (18 lines):
    Traceback (most recent call last):
      File "/tmp/tmpj_84ryar", line 280, in <module>
        main()
      File "/tmp/tmpj_84ryar", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/tmp/tmpj_84ryar", line 114, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/tmp/pip-build-env-7ovsmmjr/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
        return self._get_build_requires(
      File "/tmp/pip-build-env-7ovsmmjr/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
        self.run_setup()
      File "/tmp/pip-build-env-7ovsmmjr/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/tmp/pip-build-env-7ovsmmjr/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 8, in <module>
        import torch
    ModuleNotFoundError: No module named 'torch'
    ----------------------------------------
  WARNING: Discarding file:///home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz. Command errored out with exit status 1: /home/dtch1997/anaconda3/envs/entity-incubator/bin/python /tmp/tmpj_84ryar get_requires_for_build_wheel /tmp/tmpeaj8eltm Check the logs for full command output.
  ERROR: Command errored out with exit status 1: /home/dtch1997/anaconda3/envs/entity-incubator/bin/python /tmp/tmpj_84ryar get_requires_for_build_wheel /tmp/tmpeaj8eltm Check the logs for full command output.
dtch1997 commented 2 years ago

I re-tried the double poetry install in a fresh virtualenv and I got the CUDA version mismatch error again

Package operations: 25 installs, 0 updates, 0 removals

  • Installing torch-scatter (2.0.9): Failed

  EnvCommandError

  Command ['/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/pip', 'install', '--no-deps', 'file:///home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output:
  Processing /home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Building wheels for collected packages: torch-scatter
    Building wheel for torch-scatter (setup.py): started
    Building wheel for torch-scatter (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-6umkdi71
         cwd: /tmp/pip-req-build-p3gsf54z/
    Complete output (56 lines):
    /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
      warnings.warn(
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/utils.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/scatter.py -> build/lib.linux-x86_64-3.8/torch_scatter
    creating build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-p3gsf54z/setup.py", line 83, in <module>
        setup(
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
        return distutils.core.setup(**attrs)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
        self.run_command('build')
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 404, in build_extensions
        self._check_cuda_version()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 781, in _check_cuda_version
        raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
    RuntimeError:
    The detected CUDA version (11.4) mismatches the version that was used to compile
    PyTorch (10.2). Please make sure to use the same CUDA versions.

    ----------------------------------------
    ERROR: Failed building wheel for torch-scatter
    Running setup.py clean for torch-scatter
  Failed to build torch-scatter
  Installing collected packages: torch-scatter
      Running setup.py install for torch-scatter: started
      Running setup.py install for torch-scatter: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-r_me8lcv/install-record.txt --single-version-externally-managed --compile --install-headers /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/include/site/python3.8/torch-scatter
           cwd: /tmp/pip-req-build-p3gsf54z/
      Complete output (60 lines):
      /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/utils.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/scatter.py -> build/lib.linux-x86_64-3.8/torch_scatter
      creating build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      running build_ext
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-req-build-p3gsf54z/setup.py", line 83, in <module>
          setup(
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
          return distutils.core.setup(**attrs)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/install.py", line 545, in run
          self.run_command('build')
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 404, in build_extensions
          self._check_cuda_version()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 781, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (11.4) mismatches the version that was used to compile
      PyTorch (10.2). Please make sure to use the same CUDA versions.

      ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p3gsf54z/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-r_me8lcv/install-record.txt --single-version-externally-managed --compile --install-headers /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/include/site/python3.8/torch-scatter Check the logs for full command output.
vwxyzjn commented 2 years ago

The following is my installation trace. By comparing it with yours, it seems that you are not using a virtualized environment, which may be the cause of the issue.

Mine was Command ['/Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/pip' and yours was Command ['/home/dtch1997/anaconda3/envs/entity-incubator/bin/python'.

Alternatively, could you check if torch is installed? It is also possible that yours tried to install torch-scatter first. See https://asciinema.org/a/Gb617Fd5LmwZZCjuv4CNtZt52

 • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing torch (1.10.0): Installing...
  • Installing torch-scatter (2.0.9): Failed

  EnvCommandError

  Command ['/Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/pip', 'install', '--no-deps', 'file:///Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output: 
  Processing /Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"'; __file__='"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-pip-egg-info-ca0l2eja
         cwd: /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py", line 8, in <module>
  • Installing torch-scatter (2.0.9): Failed

  EnvCommandError

  Command ['/Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/pip', 'install', '--no-deps', 'file:///Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output: 
  Processing /Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"'; __file__='"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-pip-egg-info-ca0l2eja
         cwd: /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py", line 8, in <module>
  • Installing torch (1.10.0)
  • Installing torch-scatter (2.0.9): Failed

  EnvCommandError

  Command ['/Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/pip', 'install', '--no-deps', 'file:///Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output: 
  Processing /Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/costahuang/Library/Caches/pypoetry/virtualenvs/incubator-EKBuw-J_-py3.9/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"'; __file__='"'"'/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-pip-egg-info-ca0l2eja
         cwd: /private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/cl/tr80wk7j67q901vsnr9qn7740000gn/T/pip-req-build-l1s52nf9/setup.py", line 8, in <module>
        import torch
    ModuleNotFoundError: No module named 'torch'
    ----------------------------------------
  WARNING: Discarding file:///Users/costahuang/Library/Caches/pypoetry/artifacts/74/20/d3/78a7c5717bdeac6c1001384d14c931e314705bef64099facec1af1eb18/torch_scatter-2.0.9.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

  at ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│ 
      1103│         return decode(output)
      1104│ 
      1105│     def execute(self, bin, *args, **kwargs):

  • Installing werkzeug (2.0.2)
  • Installing yaspin (2.1.0)
costahuang@Costas-Mac-mini incubator % poetry install
Installing dependencies from lock file

Package operations: 25 installs, 0 updates, 0 removals

  • Ins  • Installing torch-scatter (2.0.9)
  • Installing attrs (21.2.0)
  • Installing iniconfig (1.1.1)
  • Installing ipython (7.30.1)
  • Installing msgpack-numpy (0.4.7.1)
  • Installing mypy-extensions (0.4.3)
  • Installing packaging (21.3)
  • Installing pathspec (0.9.0)
  • Installing platformdirs (2.4.0)
  • Installing pluggy (1.0.0)
  • Installing py (1.11.0)
  • Installing regex (2021.11.10)
  • Installing tensorboard (2.7.0)
  • Installing toml (0.10.2)
  • Installing tomli (1.2.2)
  • Installing tqdm (4.62.3)
  • Installing wandb (0.12.7)
  • Installing enn-zoo (0.1.0 /Users/costahuang/go/src/github.com/vwxyzjn/incubator/enn_zoo)
  • Installing entity-gym (0.1.0 /Users/costahuang/go/src/github.com/vwxyzjn/incubator/entity_gym)
  • Installing rogue-net (0.1.0 /Users/costahuang/go/src/github.com/vwxyzjn/incubator/rogue_net)
  • Installing black (21.11b1)
  • Installing ipdb (0.13.9)
  • Installing mypy (0.910)
  • Installing pytest (6.2.5)
  • Installing enn-ppo (0.1.0 /Users/costahuang/go/src/github.com/vwxyzjn/incubator/enn_ppo)
dtch1997 commented 2 years ago

Judging by my stack trace it's the first issue and not the second:

...
  • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing torch (1.10.0): Installing...
  • Installing torch (1.10.0)
  • Installing torch-scatter (2.0.9): Failed

I see, so there may be some undesirable interaction between Poetry and Anaconda in this case. I've always used conda together with poetry, @vwxyzjn how should I use poetry without it?

dtch1997 commented 2 years ago

I re-tried the double install outside of my conda env. This time we can see it's using the virtualenv correctly as the command is Command ['/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/pip, however I got the CUDA version mismatch error again. Any suggestions on how to fix that? 

dtch1997@DESKTOP-AR4R24K:~/code/incubator$ poetry install
Installing dependencies from lock file

Package operations: 25 installs, 0 updates, 0 removals

  • Installing torch-scatter (2.0.9): Failed

  EnvCommandError

  Command ['/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/pip', 'install', '--no-deps', 'file:///home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output:
  Processing /home/dtch1997/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Building wheels for collected packages: torch-scatter
    Building wheel for torch-scatter (setup.py): started
    Building wheel for torch-scatter (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-179lp234/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-179lp234/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-2l3lui4u
         cwd: /tmp/pip-req-build-179lp234/
    Complete output (56 lines):
    /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
      warnings.warn(
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/utils.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter
    copying torch_scatter/scatter.py -> build/lib.linux-x86_64-3.8/torch_scatter
    creating build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-179lp234/setup.py", line 83, in <module>
        setup(
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
        return distutils.core.setup(**attrs)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
        self.run_command('build')
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 404, in build_extensions
        self._check_cuda_version()
      File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 781, in _check_cuda_version
        raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
    RuntimeError:
    The detected CUDA version (11.4) mismatches the version that was used to compile
    PyTorch (10.2). Please make sure to use the same CUDA versions.

    ----------------------------------------
    ERROR: Failed building wheel for torch-scatter
    Running setup.py clean for torch-scatter
  Failed to build torch-scatter
  Installing collected packages: torch-scatter
      Running setup.py install for torch-scatter: started
      Running setup.py install for torch-scatter: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-179lp234/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-179lp234/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3_ja_mk5/install-record.txt --single-version-externally-managed --compile --install-headers /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/include/site/python3.8/torch-scatter
           cwd: /tmp/pip-req-build-179lp234/
      Complete output (60 lines):
      /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/utils.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter
      copying torch_scatter/scatter.py -> build/lib.linux-x86_64-3.8/torch_scatter
      creating build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.8/torch_scatter/composite
      running build_ext
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-req-build-179lp234/setup.py", line 83, in <module>
          setup(
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
          return distutils.core.setup(**attrs)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/install.py", line 545, in run
          self.run_command('build')
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/dtch1997/anaconda3/envs/mlops/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 404, in build_extensions
          self._check_cuda_version()
        File "/home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 781, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (11.4) mismatches the version that was used to compile
      PyTorch (10.2). Please make sure to use the same CUDA versions.

      ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-179lp234/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-179lp234/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3_ja_mk5/install-record.txt --single-version-externally-managed --compile --install-headers /home/dtch1997/.cache/pypoetry/virtualenvs/incubator-aS8zkrA6-py3.8/include/site/python3.8/torch-scatter Check the logs for full command output.

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│
      1185│         return decode(output)
      1186│
      1187│     def execute(self, bin, *args, **kwargs):
vwxyzjn commented 2 years ago

I see. You can try installing torch compiled with a newer version of cuda. You can try overriding the enn_ppo/pyproject.toml, but it's probably easier to just do the pip override

poetry run pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
poetry install
dtch1997 commented 2 years ago

What worked for me in the end was to delete the torch and torch-scatter requirements from enn_ppo/pyproject.toml and rogue_net/pyproject.toml and install them manually. 

# Delete torch, torch-scatter, then run: 
poetry install
poetry run pip install torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
poetry run pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
cswinter commented 2 years ago

Perhaps that should be the default way to install, I believe there's currently no good way to automatically choose the right CUDA version in poetry. Keeping this issue open to track updating pyproject.toml, CI, and docs.

cswinter commented 2 years ago

@vwxyzjn Would be curious to get your thoughts on what the best approach would be.

cswinter commented 2 years ago

Also, @dtch1997 thanks for the sleuthing, having a seamless setup process is super important, and smoothing out any kinks will make it much easier for new contributors to get ramped up.

vwxyzjn commented 2 years ago

This is a little bit unfortunate where some of torch's version is not on PyPi such as the torch==1.10.0+cu113. A less ideal solution is to have folks use cuda 10.2 that the PyPi's torch is built on.

Not exactly sure what would be the best solution here.

In https://poets-ai.github.io/elegy/contributing/, they use poetry install first, then use pip install to override for platform-dependent things.

vwxyzjn commented 2 years ago

After looking this further, environment markers seem to offer help in this situation (see link1, link2, link3). Gonna try looking further into this.

vwxyzjn commented 2 years ago

After thinking and playing it a bit, I think @dtch1997's approach is more preferable. It avoids many headaches at the cost of not exactly locked dependencies.

# Delete torch, torch-scatter, then run: 
poetry install
poetry run pip install torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
poetry run pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html