[0.7.11] - > Task :python:installBuildRequirements FAILED

[jaredsburrows]

Repro:

$ gradlew clean build

Error:

> Task :python:installBuildRequirements FAILED
Error installing package using `[/Users/<>/repo/cs-interview-questions/python/build/venv/bin/python, /Users/<>/repo/cs-interview-questions/python/build/venv/bin/pip, install, --disable-pip-version-check, --no-deps, --upgrade, /Users/<>/.gradle/caches/modules-2/files-2.1/pypi/mccabe/0.5.2/14c5b2794cae2a723cdf4ce520c4e0d8cae62986/mccabe-0.5.2.tar.gz]`
Processing /Users/<>/.gradle/caches/modules-2/files-2.1/pypi/mccabe/0.5.2/14c5b2794cae2a723cdf4ce520c4e0d8cae62986/mccabe-0.5.2.tar.gz
    Complete output from command python setup.py egg_info:
    Couldn't find index page for 'pytest-runner' (maybe misspelled?)
    No local packages or working download links found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/1w/20_t1jyx6ggfdscxxg8gq0gw0000gn/T/pip-3RHXIR-build/setup.py", line 57, in <module>
        'Topic :: Software Development :: Quality Assurance',
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/setuptools/dist.py", line 317, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/setuptools/dist.py", line 372, in fetch_build_eggs
        replace_conflicting=True,
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 846, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1118, in best_match
        return self.obtain(req, installer)
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1130, in obtain
        return installer(requirement)
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/setuptools/dist.py", line 440, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Users/<>/repo/cs-interview-questions/python/build/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 668, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/1w/20_t1jyx6ggfdscxxg8gq0gw0000gn/T/pip-3RHXIR-build/

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':python:installBuildRequirements'.
> Failed to install mccabe-0.5.2. Please see above output for reason, or re-run your build using ``gradle -i build`` for additional logging.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 24s
31 actionable tasks: 30 executed, 1 up-to-date

[ameliedagenais]

I've been running into the same issue. If you look at the setup.py inside https://linkedin.jfrog.io/linkedin/pypi-external/pypi/mccabe/0.5.2/mccabe-0.5.2.tar.gz it defines a dependency on pytest-runner through setup_requires=['pytest-runner']. Unfortunately, that's bad practice because it will then try to resolve this dependency during installation and there's no way to specify where to locate this dependency. (See https://pip.readthedocs.io/en/1.4.1/cookbook.html#controlling-setup-requires). It seems like this location has been broken or unreliable lately.

I was able to workaround this issue by using the mccabe-0.5.2 wheel intead since it won't run through the setup.py the same way. I had to publish it to my own repo. Here's what I did:

1. I downloaded the the wheel file from https://pypi.python.org/packages/17/9c/66792b5f917a09f7e433dfd6e20ac12964006e1d794f799c2333afc10be1/mccabe-0.5.2-py2.py3-none-any.whl#md5=a8d6b93fd1a93478f39076cb0fc4d65c
2. I manually updated the ivy.xml to point to a wheel instead. From https://linkedin.jfrog.io/linkedin/pypi-external/pypi/mccabe/0.5.2/mccabe-0.5.2.ivy, I updated the publications to:

   <publications>
   <artifact name="mccabe" type="whl" ext="whl" conf="default" m:classifier="py2.py3-none-any"  />
   </publications>

3. I uploaded both to my own repo
4. I updated my build.gradle file to point to my own repo before PyGradlePyPi:

repositories {
    ivy {
        url 'myInternalRepo'
        layout 'pattern', {
            artifact '[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]'
            ivy '[module]/[revision]/[module]-[revision].ivy'
            m2compatible = true
        }
    }
    pyGradlePyPi()
}

I hope this can help you get unblocked. Nonetheless, I think the mccabe-0.5.2 package should be fixed to not use setup_requires. Or another option would be to update PyGradlePyPi to use a wheel package instead of a tar.gz for mccabe-0.5.2.

[zvezdan]

@ameliedagenais The setup_requires is sometimes necessary -- it's not a bad practice, it's just that pip does not process it, but rather only setuptools.

@jaredsburrows I don't know the particular details of your environment, but perhaps the external network access is not allowed so setuptools couldn't reach pypi.python.org, or some network connection transient issue happened.

We actually provide a feature in pygradle that helps with this. It's the optional input named distutilsCfg on the InstallVirtualEnvironment task.

In a typical enterprise, the external network access would not be allowed on build hosts. All used Python packages would have to be in an internal PyPI mirror. But then, there must be a way to tell somehow setuptools where to find packages it processes for setup_requires. The feature we provided allows you to define an entry for distutils.cfg file like this:

public static final String DISTUTILS_CFG = """\
        | [easy_install]
        | allow_hosts = *.example.com*
        | index_url = https://something.example.com/path/to/pypi/simple
        | no_find_links = true
        """.stripMargin().stripIndent()

and then use it in the provided input in a project.afterEvaluate closure :

/*
 * Customize distutils.cfg inside the virtualenv to ensure all the
 * packages resolve setup_requires with the same restrictions.
 * This is needed because pip does not resolve setup_requires, setuptools does.
 * See the docs for virtualenv at https://virtualenv.pypa.io and pip at
 * https://pip.pypa.io/en/stable/reference/pip_install/#controlling-setup-requires
 */
project.tasks.withType(InstallVirtualEnvironmentTask) { InstallVirtualEnvironmentTask task ->
    task.distutilsCfg = DISTUTILS_CFG
}

[jaredsburrows]

@zvezdan I had trouble building with the latest version from my home computer. The previous version works fine.

[zvezdan]

@jaredsburrows It's one of those unpredictable transient errors in talking to PyPI. It seems to be happening more recently because of the transition from pypi.python.org to pypi.org (Warehouse).

There are ways to work around it. For a small project, you can just add it into your dependencies under setupRequires configuration (not python runtime dependencies):

dependencies {
    setupRequires 'pypi:pytest-runner:2.9'
    // ...
}

That way Gradle will fetch it and install it, so setuptools will not need to contact PyPI for it.

For a larger project or group of projects you may want to build a small plugin that in its apply method has project.plugins.apply(PythonPlugin) to do standard pygradle application. Then you could add in the rest of that method.

import com.linkedin.gradle.python.util.ExtensionUtils
// ...
import static com.linkedin.gradle.python.util.StandardTextValues.CONFIGURATION_SETUP_REQS
// ...
def pythonExtension = ExtensionUtils.getPythonExtension(project)
// ...
pythonExtension.forceVersion('pypi', 'pytest-runner', '2.9')
// ...
project.dependencies.add(CONFIGURATION_SETUP_REQS.getValue(), pythonExtension.forcedVersions['pytest-runner'])
// ...

That way your plugin controls this automatically for all your projects/users.

[anjaneyulu1989]

@zvezdan / @jaredsburrows, Am a new bee to pygradle, Facing the same issue on example-project build.

build.gradle

plugins { id "com.linkedin.python" version "0.7.14" } dependencies { python 'pypi:requests:2.9.1' test 'pypi:mock:1.3.0' } repositories { pyGradlePyPi() }

Could someone give me the steps to fix this.

[zvezdan]

See my suggestion about setupRequires above @anjaneyulu1989 please.

We also added pytest-runner-2.9 and setuptools-scm-1.15.5 to our pypi-external repo which should help as well.

Due to the move of PyPI from pypi.python.org to pypi.org, though, we'll also go through the code and release a new version with that URL corrected.

[yasz]

@zvezdan please help to rename setuptools_scm-1.15.5.tar.gz to setuptools-scm-1.15.5.tar.gz in linkedin jfrog. Thank you!

[maksimlikharev]

yes, it is incorrect artifact

Could not find setuptools-scm.tar.gz (pypi:setuptools-scm:1.15.5). Searched in the following locations: https://linkedin.jfrog.io/linkedin/pypi-external/pypi/setuptools-scm/1.15.5/setuptools-scm-1.15.5.tar.gz

setuptools-scm-1.15.5.ivy 27-Apr-2018 22:37 588 bytes setuptools_scm-1.15.5.tar.gz 27-Apr-2018 22:37 25.15 KB

note - and _

[anjaneyulu1989]

Hi @zvezdan , No luck even after adding the following, Is there anything am missing here.

dependencies { setupRequires 'pypi:pytest-runner:2.9' // ... }

If we already fixed this, Can you please give me the fix version of linkedin pygradle.

[zvezdan]

@maksimlikharev and @yasz The artifact name is correct. The dependency asks for setuptools-scm and that will look into the Ivy file with that name. The Ivy file contains the correct name of the artifact:

<artifact name="setuptools_scm" ext="tar.gz" conf="default" type="tar.gz" />

Thus, Gradle is able to find the artifact. It's the same in our repo at LinkedIn and builds work just fine. Please, make sure to provide setupRequires ... dependency for setuptools-scm if you continue experiencing the issue with that package.

Also, both python tooling and PyGradle tolerate correctly both naming conventions and convert between them as needed.

[zvezdan]

@anjaneyulu1989 There are newer releases of PyGradle. Try 0.8.10 or 0.8.11 -- I'm sure they are available on plugins.gradle.org. I'll release the latest in 0.9.x series in a couple of hours too.