Error installing signatory and torch simultaneously

[atimermans]

Hi. I'm trying to install signatory, along torch with:

pip install -r requirements.txt

Being said requirements.txt:

torch==1.7.1
signatory==1.2.4.1.7.1

And I get the following error:

Collecting torch==1.7.1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/90/4f/acf48b3a18a8f9223c6616647f0a011a5713a985336088d7c76f3a211374/torch-1.7.1-cp36-cp36m-manylinux1_x86_64.whl (776.8MB)
    100% |████████████████████████████████| 776.8MB 112kB/s 
Collecting signatory==1.2.4.1.7.1 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/58/93/cea761d492f908133d513fb1f93eb57bd0839b8ecd9eda0e97848a1c3c8f/signatory-1.2.4.1.7.1.tar.gz (61kB)
    100% |████████████████████████████████| 61kB 32.0MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/tmp/pip-install-fcnz7oau/signatory/setup.py", line 21, in <module>
        import torch.utils.cpp_extension as cpp
    ModuleNotFoundError: No module named 'torch'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-fcnz7oau/signatory/setup.py", line 23, in <module>
        raise ImportError("PyTorch is not installed, and must be installed prior to installing Signatory.")
    ImportError: PyTorch is not installed, and must be installed prior to installing Signatory.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-fcnz7oau/signatory/

Is there any way to install torch and signatory simultaneously? This is work-related, and deployments are automated, so it would be kind of hard to make an exception for the service I'm developing so that it runs two different pip installs

I'm using Python 3.6.13 with pip==18.1 (although I tried it with pip==21.0.1 and get the same error). I'm using Ubuntu, the result of lsb_release -a is:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

Thank you very much in advance.

[patrick-kidger]

So in theory I think it should be possible to update Signatory to make this work.

We'd need to include a pyproject.toml file, which can specify that torch is necessary to install Signatory. The version of PyTorch it specifies should carefully match the version of Signatory that's being downloaded -- i.e. the contents of pyproject.toml must be version-specific, and created by Signatory's build system to match the version of PyTorch it is being built against.

That's a reasonable amount of faff to set up, which is why I've never done it. Usually I do just recommend running pip install twice -- once for PyTorch, once for Signatory -- and that covers 99% of use cases. (My general experience has been that setting up an environment usually involves more than a single pip install, so lots of devs expect to be able to run arbitrary setup scripts, rather than being limited to a single requirements.txt file.)

If you can I'd suggest finding a way to run pip install twice; once for PyTorch and once for Signatory.

If that's really not possible for you, then I would be happy to accept a PR, making the necessary changes to Signatory's build system, that does what I outlined above. Realistically this isn't something I've got time to set up myself, I'm afraid.

[Xylambda]

So, if I undestood correctly, a pyproject.toml file with the following content should be added:

# These are the assumed default build requirements from pip:
# https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
# About versioneer pip<19:
# https://github.com/warner/python-versioneer/issues/192#issuecomment-623144826
requires = ["setuptools>=40.8.0", "wheel", "versioneer-518", "torch<=1.7.1"]
build-backend = "setuptools.build_meta"

I'm not sure if a follow the versions part: wouldn't be enough to specify an upper bound for the torch version?

On the other hand, I think the command pip install torch will only install the cpu version. What would be the correct way to handle the GPU support?

[patrick-kidger]

I don't think that'll be enough.

If that's used, and someone does:

pip install signatory==1.2.4.1.6.0
pip install torch==1.6.0

Then the first pip will grab Signatory, look for PyTorch, and download the most recent compatible version (1.7.1). It will then compile Signatory against that version. Then the second install will downgrade PyTorch, leaving it incompatible with Signatory.

We instead need to specify a different pyproject.toml for each version of PyTorch that Signatory is built against, with each one specifying the correct requires = ["torch==<version>"].

[Xylambda]

I see. We need to discuss this internally because we may prefer to handle this issue in our own CICD (by calling some .sh file to force torch installation first, for example).

Anyway, I would like to solve the issue in its origin instead of on our pipeline, so I'm personally open to create a PR .