No longer able to build from source with `pip install -v .`

[xwang233]

🐛 Describe the bug

After PR https://github.com/pytorch/vision/pull/4384, I'm no longer able to build torchvision with

pip install -v .

, though

python setup.py develop

still works.

Is this the intended behavior now?

The pip install -v . now reports error message

xxx@yyy:/opt/pytorch/vision# pip install -v .
Using pip 21.2.2 from /opt/conda/lib/python3.8/site-packages/pip (python 3.8)
Processing /opt/pytorch/vision
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Running command /opt/conda/bin/python /tmp/pip-standalone-pip-qzzditq9/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-klifg95l/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
  Collecting setuptools>=40.8.0
    Using cached setuptools-58.2.0-py3-none-any.whl (946 kB)
  Collecting wheel
    Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel, setuptools
  Successfully installed setuptools-58.2.0 wheel-0.37.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing build dependencies ... done
  Running command /opt/conda/bin/python /opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmptw3tr8z6
  Traceback (most recent call last):
    File "/opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-klifg95l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-klifg95l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-klifg95l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-klifg95l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 12, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'
  Getting requirements to build wheel ... error
WARNING: Discarding file:///opt/pytorch/vision. Command errored out with exit status 1: /opt/conda/bin/python /opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmptw3tr8z6 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /opt/conda/bin/python /opt/conda/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmptw3tr8z6 Check the logs for full command output.

Versions

I'm using the latest pytorch version 1.11.0a0+git458a00b https://github.com/pytorch/pytorch/commit/458a00bacb68998a75a87af153e6ae3690698bc2,

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.19.6
Libc version: glibc-2.31

Python version: 3.8.11 (default, Aug  3 2021, 15:09:35)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.17
Is CUDA available: True

cc @ptrblck @pmeier @NicolasHug

[NicolasHug]

We don't recommend using pip, but if you prefer pip over setup.py, you'll now need to use the --no-build-isolation flag:

pip install --no-build-isolation -v .

The reason this happens is because of the new pyproject.toml file introduced in #4384, which is needed to configure black.

pyproject.toml is a file that is supposed to specify things like build-time dependencies. The existence of this file in the root directory is enough for pip to change its default behavior, as documented here. This change of behavior breaks the torchvision build.

For a reason that is beyond my understanding, this is the file that black and other tools have chosen as their config file (and they have no plan on supporting anything else unfortunately https://github.com/psf/black/issues/683)

So yes, it's absurd, but introducing black in a repo will break pip :)

[xkszltl]

Here's another victim. I was aware of similar issues previously so it only wasted me ~1h to figure out what's going on. The error msg wasn't clear for sure.

Since torchvision is introducing pyproject.toml, how about simply make it complete?

Even more surprisingly, the issue only exists in a CentOS 7 docker, but Ubuntu 18.04 with similar setup doesn't have the issue at all.

[xkszltl]

Adding --no-build-isolation or use setup.py directly isn't always possible, especially if it's in a big pipeline that's hard to expose every details to be configurable for each project. My workaround is to clone the repo, rm -f pyproject.toml, and pip install from local.