build issue - createVirtualEnvironment -> pygradleBootstrap

mjothiram commented 3 years ago

I see this issue in my build from this morning. And trying to see this PyPi in LinkedIn Artifactory, but could not see that. Is it removed or any issue?

Thanks

Execution failed for task ':airflow:createVirtualEnvironment'.

Could not resolve all files for configuration ':airflow:pygradleBootstrap'. Could not resolve pypi:virtualenv:16.0.0. Required by: project :airflow Could not resolve pypi:virtualenv:16.0.0. Could not get resource 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'. Could not GET 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'. Received status code 401 from server: Unauthorized

rafishaiks03 commented 3 years ago

Hi

Can anyone please address the above issue, iam also facing the same.

I'm not able to get a solution....

Thanks in Advance..!!

sudhir418 commented 3 years ago

Even we are facing the same issue , any alternatives?

byblakeorriver commented 3 years ago

Same here.

rafishaiks03 commented 3 years ago

I'm still facing the same issue and didn't find any alternatives also.. still looking for help.. if not dropping Pygradle is the only option left...

griffinptg commented 3 years ago

I was able to resolve by downloading dependencies from pypi and uploading to my company's Artifactory. I had to make an .ivy file for each. It was a pain!

For example: Download the .tar.gz here: https://pypi.org/project/virtualenv/16.0.0/ Upload that to your own Artifactory (assuming you have one) with an ivy file, virtualenv-16.0.0.ivy:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="virtualenv" revision="16.0.0" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="virtualenv" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default" />
</ivy-module>

There are a bunch more after this FYI. And if you get a missing Python import instead of the 401, do the same, but also add it as a dependency to the lib that needs it, for example:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="requests" revision="2.18.4" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="requests" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default">
    <dependency org="pypi" name="urllib3" rev="1.26.6" conf="default" />
    <dependency org="pypi" name="chardet" rev="4.0.0" conf="default" />
    <dependency org="pypi" name="certifi" rev="2021.5.30" conf="default" />
    <dependency org="pypi" name="idna" rev="3.2" conf="default" />
  </dependencies>
</ivy-module>

Just posting this as a workaround. It would still be good to see a fix for this.

rafishaiks03 commented 3 years ago

Hi @griffinmm,

It seems you have posted in maven dependency format could you please help me out how and where to add the above code in gradle build. May be you can share your build.gradle file with the above things.

Thanks in Advance...!

griffinptg commented 3 years ago

@rafishaiks03 Here's my build.gradle, sanitized a bit:

buildscript {
    repositories {
        ivy {
            url 'http://artifactory.yourdomain.com/artifactory/plugins-release'
            credentials {
                username = "${artifactory_user}"
                password = "${artifactory_password}"
            }
        }
    }
    dependencies {
        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.15.1"
    }
}

plugins {
    id 'com.linkedin.python' version '0.9.11'
}

allprojects {
    apply plugin: "com.jfrog.artifactory"
}

artifactory {
    contextUrl = "https://artifactory.yourdomain.com/artifactory"
    resolve {
        repository {
            repoKey = 'pypi-local'
            username = "${artifactory_user}"
            password = "${artifactory_password}"
            maven = false
            ivy {
                ivyLayout =  "[organisation]/[module]/[revision]/[module]-[revision].ivy"  
                artifactLayout =  "[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
                mavenCompatible = false
            }
        }
    }
}

python {
   testDir = project.file('integration-test').path

   pythonEnvironment = loadEnvFile()

   details {
       pythonVersion = '3.6'
   }

   forceVersion('pypi', 'flake8', '3.2.1')
}

repositories {
   pyGradlePyPi()
}

pytest {
  additionalArguments = ["--junitxml=test-results/results.xml"]
}

dependencies {
  setupRequires 'pypi:jedi:0.11.1'
  setupRequires 'pypi:autopep8:1.3.3'
  setupRequires 'pypi:rope:0.10.7'
  setupRequires 'pypi:yapf:0.20.0'

  test 'pypi:requests:2.18.4'
  test('pypi:boto3:1.5.18') {
    exclude module: 'docutils'
  }
}

def loadEnvFile() {
  def m = [:]
  if(project.hasProperty('configFile')) {
    file('integration-test/config/' + configFile).eachLine { line ->
      def tokens = line.tokenize('=')
      println "Setting property: " + tokens
      m.put(tokens[0], tokens[1])
    }
  }
  return m
}

alghoshal commented 3 weeks ago

For anyone else facing these pygradle issue, been able to build on Python-3.8. Like @griffinmm had to download modules from pypi, with some differences:

Downloaded them using the pivy-importer (see: https://github.com/linkedin/pygradle.git), which also prepares the ivy files automatically
Downloaded modules to a local folder (specified in build.gradle) & used that as a local repository in place of Artifactory.
Had to fix issues with name, case, dependencies, etc in some of modules (see Script-2 below: fixDownloadedIvyModules.sh)

Details shared here: Project: https://github.com/alghoshal/pygradle_python3_example/blob/main/README.md

Build file: https://github.com/alghoshal/pygradle_python3_example/blob/main/build.gradle

Script-1 (Downloader): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/downloadPyGradleDependecies.sh

Script-2 (Fixes name, etc): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/fixDownloadedIvyModules.sh

linkedin / pygradle

build issue - createVirtualEnvironment -> pygradleBootstrap #351