ERROR during the pip installation process

[lwwlwwl]

I was trying to install pip and

ERROR: Could not find a version that satisfies the requirement torch (from marius==0.0.2) (from versions: none)
ERROR: No matching distribution found for torch (from marius==0.0.2)

keeps popping up.

Tried to pip install torchvision==0.1.8 in command line and it showed Successfully installed torch-1.11.0 torchvision-0.1.8. Then, when I tried to pip3 install . again, the same error appears. I am wondering how to solve this to proceed. Thank you.

[JasonMoho]

This could be a difference between pip and pip3, causing things to install in different locations. pip3 install torch might fix it for you.

You can check if torch is installed with: python3 -c "import torch; print(torch.__version__)"

Also see the pytorch docs for specific install commands for your deployment

[lwwlwwl]

pip3 install torch gives the same error. I followed the link trying to install PyTorch via Anaconda, run pip3 install . and it gives a new error saying it may not be a problem with pip.

       note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for numpy
      Failed to build numpy
      ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

torch is installed with v1.10.1

[JasonMoho]

Hard to say what's wrong here without more information about your environment.

What is your:

• Operating system
• pip version
• python version

[JasonMoho]

To get the above information you can run this script: python3 src/python/distribution/marius_env_info.py

You may have to install importlib_metadata first: pip3 install importlib_metadata

You should see output like the following if all works properly

python3 src/python/distribution/marius_env_info.py 
cmake:
  version: 3.21.2
cpu_info:
  num_cpus: 12
  total_memory: 32GB
cuda:
  version: N/A
gpu_info: []
marius:
  bindings_installed: false
  install_path: N/A
  version: N/A
openmp:
  version: N/A
operating_system:
  platform: macOS-11.6-x86_64-i386-64bit
pybind:
  PYBIND11_BUILD_ABI: _cxxabi1002
  PYBIND11_COMPILER_TYPE: _clang
  PYBIND11_STDLIB: _libcpp
python:
  compiler: Clang 13.0.0 (clang-1300.0.29.30)
  deps:
    breathe_version: 4.30.0
    numpy_version: 1.21.2
    omegaconf_version: 2.2.1
    pandas_version: 1.3.3
    pip_version: 22.0.4
    pyspark_version: 3.2.0
    pytest_version: 6.2.5
    sphinx_rtd_theme_version: 1.0.0
    torch_version: !!python/object/new:torch.torch_version.TorchVersion
      - 1.11.0
    tox_version: 3.24.3
  version: 3.9.12
pytorch:
  install_path: /usr/local/lib/python3.9/site-packages/torch
  version: !!python/object/new:torch.torch_version.TorchVersion
    - 1.11.0

[lwwlwwl]

For some reason python3 src/python/distribution/marius_env_info.py gives ModuleNotFoundError: No module named 'yaml'. So I ran the script directly and gave the following

/bin/sh: cmake: command not found
clang: error: unsupported option '-fopenmp'
cmake:
  version: N/A
cpu_info:
  num_cpus: 2
  total_memory: 8GB
cuda:
  version: N/A
gpu_info: N/A
marius:
  bindings_installed: false
  install_path: N/A
  version: N/A
openmp:
  version: N/A
operating_system:
  platform: macOS-10.16-x86_64-i386-64bit
pybind:
  PYBIND11_BUILD_ABI: _cxxabi1002
  PYBIND11_COMPILER_TYPE: _clang
  PYBIND11_STDLIB: _libcpp
python:
  compiler: 'Clang 10.0.0 '
  deps:
    breathe_version: N/A
    numpy_version: 1.19.2
    omegaconf_version: N/A
    pandas_version: 1.1.3
    pip_version: 20.2.4
    pyspark_version: N/A
    pytest_version: 6.1.1
    sphinx_rtd_theme_version: N/A
    torch_version: !!python/object/new:torch.torch_version.TorchVersion
      - 1.10.1
    tox_version: N/A
  version: 3.8.5
pytorch:
  install_path: /Users/yuchen/opt/anaconda3/lib/python3.8/site-packages/torch
  version: !!python/object/new:torch.torch_version.TorchVersion
    - 1.10.1

For cmake here, I installed the extension CMake Tools in vscode but it seems not working. For Clang, I tried Clang --version and it gives

Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[JasonMoho]

One thing I would try is to create a new python virtual environment and do the pip3 install process again:

$ python3 -m venv my_env
$ source my_env/bin/activate
(my_env) $ pip3 install torch 
(my_env) $ pip3 install . 

It's possible that the mixture of different installation methods (conda, pip, vscode) have put your dependencies in a weird state, so starting fresh may help.

Also I found this stack overflow post which has the same numpy error message, might be worth checking brew doctor to see if your Xcode tools are up to date.

[lwwlwwl]

Many thanks for your help. I tried new env and it didn't work(giving the same error). brew doctor showed Xcode tools were too outdated, so I updated it as said in the post, which didn't work either(the same error).

Then, I tried to downgrade python from v3.11 to v3.9 (and installed cmake in local computer in addition to vscode extension), which solved the problem of numpy. However, a new error came up subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'bindings', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2. I'm wondering if this comes from cmake not being installed properly.

[JasonMoho]

Was there more to the error message?

I just noticed that in your output of marius_env_info that the reported clang version used to build python is 10.0, but the output of Clang --version is 12.0. We have an explicit check in our build script requiring the clang version to be >= 11.0.

You can check the version of clang used to build python like the following:

$ python3 -c "import sys; print(sys.version)"
3.9.12 (main, Mar 26 2022, 15:52:10) 
[Clang 13.0.0 (clang-1300.0.29.30)]

It may be possible that pip is using clang10 causing the build to fail. If that's the case I would expect Clang version must be at least 11! to be printed out in the error message.

[lwwlwwl]

I think that might be the case. I uninstalled python and pip and reinstalled them again. Then, pip3 install . gave ERROR: Failed building wheel for marius but showed Successfully installed marius-0.0.2 at the end. So, I assume marius is successfully installed using an alternative. But then, import Marius gives a seg fault.

[JasonMoho]

Sorry for the continued difficulties. My guess is that the build of the C++ sources failed but pip continued with the installation process. I'll try to reproduce this behavior and work on a fix.

Can you try installing in verbose mode and post the full output? You can attach the output as a text file. This should tell me exactly what failed in the build.

pip3 uninstall marius 
pip3 install . --verbose

[lwwlwwl]

This is all I have in the terminal. Very first part of it seems to be cropped but I don't know to restore it. Thank you for your help. marius terminal output.txt

[lwwlwwl]

I was able to set it up and run examples on my local computer by removing one miniconda folder from user but I am getting errors when trying to install on a GPU-enabled machine.

I tried two settings (all meet the requirements listed) but there are still some errors

• For cuda 11.2/cudnn 8.1.1/pytorch 1.11.0/python 3.8.10/gcc 9.3.0/cmake 3.14.5/make 4.3, the error message is Target "cmTC_51b76" requires the language dialect "CUDA17" (with compiler extensions), but CMake does not know the compile flags to use to enable it
• For the other setting, it is python 3.9.4/cmake 3.19.4 where all the other remain the same, the error message is CMake Error at src/cpp/third_party/CMakeLists.txt:9 (message):/usr/bin/git submodule update --init dependencies/pybind11 failed with exit code 1, please checkout submodules

[rogerwaleffe]

Glad you were able to get it up and running on your local computer.

In terms of the GPU-enabled machine installation, I'm not sure what is causing those error messages. Can you try to install and run using docker according to the following steps:

1. Check if docker is installed (which docker) and if not install it: https://docs.docker.com/engine/install/
2. Check if docker can access the GPUs by running sudo docker run --gpus all nvidia/cuda:11.0-base nvidia-smi. If this doesn't print the output of nvidia-smi, docker cannot access the CUDA driver on the host machine. See here: https://docs.nvidia.com/ai-enterprise/deployment-guide/dg-docker.html#:~:text=You%20do%20not%20need%20to,devices%20in%20the%20Docker%20runtime.
3. Once the above succeeds, you should no longer need anything installed on the host machine. Add the following to a file called Dockerfile:

FROM nvidia/cuda:11.4.0-cudnn8-devel-ubuntu18.04
RUN apt update

RUN apt install -y g++ \
         make \
         wget \
         unzip \
         vim \
         git \
         dstat \
         python3-pip

# install gcc-9
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test
RUN apt update
RUN apt install -y gcc-9 g++-9
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-9 9

# install cmake 3.20
RUN wget https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.sh
RUN mkdir /opt/cmake
RUN sh cmake-3.20.0-linux-x86_64.sh --skip-license --prefix=/opt/cmake/
RUN ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake

# install pytorch
RUN python3 -m pip install torch==1.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

4. Create a docker image: docker build -t image_name:image_tag .
5. Run the docker image: docker run --gpus all -it image_name:image_tag bash
6. Once the container is running, install marius:

git clone https://github.com/marius-team/marius.git
cd marius
pip3 install .

[lwwlwwl]

Thanks for your reply. Running pip3 install . using docker does seem to work where it gives

Successfully installed GPUtil-1.4.0 Jinja2-3.0.3 MarkupSafe-2.0.1 PyYAML-6.0 Pygments-2.12.0 Sphinx-5.0.2 alabaster-0.7.12 antlr4-python3-runtime-4.9.3 attrs-21.4.0 babel-2.10.3 breathe-4.34.0 certifi-2022.6.15 charset-normalizer-2.0.12 distlib-0.3.4 docutils-0.18.1 filelock-3.4.1 imagesize-1.3.0 importlib-metadata-4.8.3 importlib-resources-5.4.0 iniconfig-1.1.1 marius-0.0.2 numpy-1.19.5 omegaconf-2.2.2 packaging-21.3 pandas-1.1.5 platformdirs-2.4.0 pluggy-1.0.0 psutil-5.9.1 py-1.11.0 py4j-0.10.9.3 pyparsing-3.0.9 pyspark-3.2.1 pytest-7.0.1 python-dateutil-2.8.2 pytz-2022.1 requests-2.27.1 six-1.16.0 snowballstemmer-2.2.0 sphinx-rtd-theme-1.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 toml-0.10.2 tomli-1.2.3 tox-3.25.0 urllib3-1.26.9 virtualenv-20.15.0 zipp-3.6.0

However, when I tried to preprocess example fb15k_237 using command marius_preprocess --dataset fb15k_237 --output_dir datasets/fb15k_237_example/, it gave error

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 783, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (docutils 0.18.1 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('docutils<0.18'), {'sphinx-rtd-theme'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/marius_preprocess", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 576, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 589, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'docutils<0.18' distribution was not found and is required by sphinx-rtd-theme

[rogerwaleffe]

Great! Things should be close to running then. This last error looks like a quick fix. Try simply installing a version of docutils less than 0.18. Inside the running docker container run:

pip3 uninstall docutils
pip3 install docutils==0.17.1

[lwwlwwl]

Thank you. It works!

I am wondering if extract_file() can deal with zip file. After running both of the given node classification and link prediction examples, I was trying to do node classification on ogbn-products. So, I took the given custom_nc_graphsage.py file and modified these two line in class MYDATASET along with some directory modifications

self.dataset_name = "ogbn_products"
self.dataset_url = "http://snap.stanford.edu/ogb/data/nodeproppred/products.zip"

But it seems that the zip file was not unzipped where error says that the .content file cannot be found. Is there any way that I can work around it?

[rogerwaleffe]

Glad it is working!

Regarding your new question, yes, extract_file() should be able to deal with the ogbn-products zip file. Potentially there are some path issues causing this error.

However, you do not need to create a custom dataset for this graph. ogbn-products is a built in dataset supported by Marius. See the following:

built in dataset documentation ogbn_products.py

Thus you can preprocess it as usual: marius_preprocess --dataset ogbn_products --output_dir datasets/ogbn_products/.

Closing this issue now that the installation has been complete.