Python: Local pip install is broken

[amotta]

Running

git clone git@github.com:scalableminds/webknossos-wrap.git
cd webknossos-wrap/python
pip install .

fails with

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-ki064w71/setup.py", line 59, in <module>
        cmdclass={"build_py": BuildPyCommand},
      File "/home/amotta/.miniconda3/envs/wkw-test/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/amotta/.miniconda3/envs/wkw-test/lib/python3.7/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/amotta/.miniconda3/envs/wkw-test/lib/python3.7/site-packages/setuptools/dist.py", line 447, in __init__
        k: v for k, v in attrs.items()
      File "/home/amotta/.miniconda3/envs/wkw-test/lib/python3.7/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/home/amotta/.miniconda3/envs/wkw-test/lib/python3.7/site-packages/setuptools/dist.py", line 735, in finalize_options
        ep.load()(self, ep.name, value)
      File "/tmp/pip-req-build-ki064w71/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/integration.py", line 17, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/tmp/pip-req-build-ki064w71/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/__init__.py", line 150, in get_version
        parsed_version = _do_parse(config)
      File "/tmp/pip-req-build-ki064w71/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/__init__.py", line 113, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
    LookupError: setuptools-scm was unable to detect version for '/tmp'.

when run from within a clean Conda environment based on Python 3.7.

Assigning to @normanrz and @philippotto based on the changelog of setup.py.

[normanrz]

Does it work if you run python setup.py install?

[amotta]

Yes, that works.

[normanrz]

Isn't that the idiomatic way to install Python packages locally? The readme also references that.

[amotta]

Hmm, I don't really know. But the documentation of Python 3.8 doesn't even mention python setup.py install.

In fact, it says that "pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers."

And pip install . is listed for "installing from local source tree: https://packaging.python.org/tutorials/installing-packages/#installing-from-a-local-src-tree

[normanrz]

This is a bug in setuptools-scm, see https://github.com/pypa/setuptools_scm/issues/357

[amotta]

The Python ecosystem is a bit foreign to me. So, all of the following could be wrong.

It seems that python setup.py install and pip install . are not equivalent:

python setup.py install seems to delegates to easy_install, which does not properly handle pre-release versions on PyPI. Right now, the latest version of NumPy available via PyPI is 1.20.0rc1, which is a pre-release version that requires at least Python 3.7. When calling python setup.py install in an environment with Python 3.6, this fails with a RuntimeError: Python version >= 3.7 required..

This seems wrong:

• The setup should not pull in pre-release versions
• The requirements should be such that no version errors occur during setup

If I understand correctly (see link above), then pip install . would not pull in the pre-release version. But pip install . throws an error because of the bug linked by Norman.

As a result, it is impossible to locally install the the wkw package, right now.

Should we

• require Python ≥3.7, or
• hard-code exact versions of all dependencies
  (to prevent python setup.py install from pulling in pre-release versions), or
• fix pip install . by getting rid of setuptools-scm?

With my limited knowledge of the Python ecosystem, I'd be in favor of the last option.

What do you think?

CC @valentin-pinkau