Pydruid 0.5.9 installation fails on Python 3.7 (with pytest ??)

[potiuk]

When we try to install released yesterday pydruid 0.5.9 on python 3.7 during our CI pipeline it fails to install. Looks like it is somewhow related to pytest runner (I guess some pytest dependency in druid moved from setup-requires to install-requires maybe?)

Example here: https://travis-ci.org/github/apache/airflow/jobs/662274100?utm_medium=notification&utm_source=github_status

Collecting pydruid>=0.4.1 (from apache-airflow[devel_ci])
  Downloading https://files.pythonhosted.org/packages/ef/0d/f670f42882dd6a6c9626af2473c57ae28def715864ed4f1bec26cf1b4632/pydruid-0.5.9.tar.gz (5.9MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-n5xrjft0/pydruid/setup.py", line 51, in <module>
        "Programming Language :: Python :: 3.7",
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 143, in setup
        _install_setup_requires(attrs)
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 138, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 687, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 783, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1066, in best_match
        return self.obtain(req, installer)
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1078, in obtain
        return installer(requirement)
      File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 743, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/local/lib/python3.7/site-packages/setuptools/installer.py", line 133, in fetch_build_egg
        wheel.install_as_egg(dist_location)
      File "/usr/local/lib/python3.7/site-packages/setuptools/wheel.py", line 103, in install_as_egg
        self._install_as_egg(destination_eggdir, zf)
      File "/usr/local/lib/python3.7/site-packages/setuptools/wheel.py", line 111, in _install_as_egg
        self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
      File "/usr/local/lib/python3.7/site-packages/setuptools/wheel.py", line 132, in _convert_metadata
        os.mkdir(destination_eggdir)
    FileExistsError: [Errno 17] File exists: '/tmp/pip-install-n5xrjft0/pydruid/.eggs/pytest_runner-5.2-py3.7.egg' 

[villebro]

@potiuk I'm hitting the same error on Python 3.7.5, based on quick googling it seems to be a problem with the way pytest-runner creates temporary directories. Can look into this later, for now I suggest checking out the 0.5.9 tag or 0.5 branch from GitHub and doing a pip install ..

[potiuk]

Yep. We did that as a workaround we added limits for pydruid. Happy to remove it when it's fixed :)

[naoyak]

Also running into this.

[mistercrunch]

FWIW, pytest predates my involvement in this repo. Happy to review code that will make us 3.7.x compatible.

[naoyak]

Could this be fixed as part of #190 by removing pytest and pytest-runner from setup.py? https://github.com/pytest-dev/pytest-runner/blob/master/README.rst#deprecation-notice

[mistercrunch]

Should we move away from pytest and onto something like nosetest? How much work would that represent?

[villebro]

Agreed, we need to deprecate pytest, although I believe nose is also on its way out (I haven't tried it personally, but apparently nose2 is the successor of nose).

[villebro]

FYI @naoyak and @potiuk : this should now be fixed in pydruid==0.5.11.

[potiuk]

Cool. Will try it :)

[potiuk]

Thanks! Tested and removed the upper bound here https://github.com/apache/airflow/pull/9965 . After we merge it to master it should automatically land in our "official" constraints file with 0.6.1 (current latest pydruid version)