Resolve ModuleNotFoundError for 'torch' when Adding torch-scatter via Poetry

[Buddies-as-you-know]

Issue Overview

When attempting to add torch-scatter (version 2.1.2) to a project using Poetry, a ModuleNotFoundError for 'torch' is encountered during the build process. This issue arises in a Poetry environment.

Error Details

The error originates from the build backend, suggesting a compatibility issue with torch-scatter and PEP 517 builds, rather than a problem with Poetry itself. The traceback indicates a missing 'torch' module, despite 'torch' being included in the project dependencies.

Current Environment

• Python version: 3.10
• Operating System: Linux (exact distribution not specified)
• PyTorch version used: torch-1.11.0+cu115-cp310-cp310-linux_x86_64.whl

Steps to Reproduce

1. Initialize a Poetry environment with Python 3.10.
2. Add PyTorch (version 1.11.0) to the project.
3. Try adding torch-scatter (version 2.1.2) using Poetry.

Suspected Cause

The problem appears to be with torch-scatter not supporting PEP 517 builds. This leads to an inability to recognize the 'torch' module during the torch-scatter build process.

Suggested Initial Actions

• Verify the compatibility of torch-scatter with PEP 517 by executing pip wheel --use-pep517 "torch-scatter (==2.1.2)".
• Ensure that the 'torch' module is correctly installed and accessible within the Poetry virtual environment.
• Consider testing different versions of torch-scatter or PyTorch to identify if the issue is version-specific.

Additional Context

• Any custom configurations or dependencies that might influence the standard build process should be examined.
• Clarification of the Poetry version in use could provide additional insights into potential compatibility issues.

[rusty1s]

Did you specify the wheel page (e.g., https://data.pyg.org/whl/torch-1.11.0+cu115.html) when installing via poetry? Otherwise, this error is currently expected since torch itself is required to build the package.

[quentinblampey]

Hello @rusty1s,

I also have issues installing torch_scatter with poetry (for the reason you mentioned), but I don't like using the wheels since I need to specify a cpu or cuda usage. I also had issue using the wheels since some were missing for macOS arm64.

If this can't be fixed with poetry, is there another dependency manager that works well with torch_scatter?

I'm open to changing poetry to another dependency manager if needed. All I want is to have a clean "pip install" of my package for users. Indeed, currently, I need to install torch_scatter separately from the rest of the installation, which is really not convenient and can lead to several issues (for instance, the latest version of torch_scatter doesn't work with torch>=2.1 on macOS ARM64).

[rusty1s]

First of all, since we upstreamed most of the functionality of torch-scatter to PyTorch directly, is it still necessary for you to use this package? Secondly, I have no problem installing torch-scatter on ARM64, and I will provide ARM64 wheels as soon as possible. Lastly, if you want CUDA/CPU independent installation, the only choice would be conda right now. However, I am still not sure I understand your concerns related to cpu/cuda usage. If you just want to follow torch conventions, you can simply use cu121 (which works on cpu devices only as well).

[quentinblampey]

Thanks @rusty1s for your answer! I checked again why I was depending on torch-scatter, and it was because of AttentionalAggregation. When providing ptr as an input, I get the error below only when torch-scatter is not installed: NotImplementedError: 'softmax' requires 'index' to be specified

Since you said that most of the functionalities are now using PyTorch directly, I tried other things, and it turns out that using index instead of ptr works nicely without the torch-scatter dependency! These are really good news, I can now remove torch-scatter from my dependencies, thanks a lot 🙂