joc0611 commented 6 years ago

Loading Gradle Inspector Requires Connection to Maven Central

Expected behavior

When setting a repository URL and fixed version without wildcards for the Gradle inspector (through detect.gradle.inspector.repository.url and detect.gradle.inspector.version, no connection to Maven Central (http://repo2.maven.org) is established.

Actual behavior

Even thoughdetect.gradle.inspector.repository.url=https://my-maven-central.url/ and detect.gradle.inspector.version=0.7.0 is set, a connection to Maven Central (http://repo2.maven.org) is established.

Steps to Reproduce

Get hub-detect from https://raw.githubusercontent.com/blackducksoftware/hub-detect/4.3.1/hub-detect/src/main/resources/hub-detect-sh

In Bash, do

$ export DETECT_LATEST_RELEASE_VERSION=4.3.1`
$ hub-detect.sh \
 --blackduck.api.token=**** \
 --blackduck.url=https://my-hub.url \
 --detect.project.name=test --detect.project.version.name=0.1 \
 --detect.gradle.inspector.repository.url=https://my-maven-central.url/ \
 --detect.gradle.inspector.version=0.7.0

Version

hub-detect version: 4.3.1 and higher

Problem description

There might be build environments, where access to Maven Central is not possible due to corporate policy. In such cases, it should be possible to download the Gradle Inspector from a mirror repository.

This was possible in hub-detect version 4.3.0 by setting the parameters detect.gradle.inspector.repository.url and detect.gradle.inspector.version as shown above.

In more recent versions, a connection to Maven Central is established in any case, causing an exception to be thrown, which eventually fails the execution of hub detect.

I think the relevant commit is ccdac43fa8297cdf7b82eb3977b682316181bd6f

Possible solutions

I think there are two possible solutions:

Do not try to resolve the version from the metadata.xml in case it is not a range (this is the behavior from 4.3.0)
Use the supplied detect.gradle.inspector.repository.url to do the version resolution

Misc

Also, I think it would be great to make the hard-coded URL configurable and use an HTTPS connection in the default value.

stavvy-akamen commented 6 years ago

@joc0611 Interesting, so you are reporting a regression then. We shall investigate.

stevebillings commented 6 years ago

Hi Johannes: After considering several challenges related to "partially online" environments (where some sites are accessible, but others are not), we've concluded that the least-confusing approach overall is to require that detect either run fully online (with access to all sites it uses), or run in airgap mode.

This case is a good example. If we make the change you're suggesting, it does solve the problem, but it would require that the user deploy not just the jar, but also the maven-metadata.xml file. And detect would then stop upgrading to the latest version of the gradle inspector even if configured to do so, a fact which may come as a surprise to some users.

So, we recommend running in airgap mode. Would that work for you?

callmepills commented 6 years ago

I am facing the same issue now in my environment. The default, hard coded metadata URL is currently blocked:

http://repo2.maven.org/maven2/com/blackducksoftware/integration/integration-gradle-inspector/maven-metadata.xml

However, the following URL is unblocked for me:

https://repo.maven.apache.org/maven2/com/blackducksoftware/integration/integration-gradle-inspector/maven-metadata.xml

Should we switch to this?

Either way, it does feel weird that the JAR and metadata could be resolved/downloaded from two different repositories. If present, I think the custom repository URL should override both.

stevebillings commented 6 years ago

ah, I see. You have a good point. We are planning to resolve this in detect 5.0.0. (In 5.0.0 the default location of the integration-gradle-inspector jar and maven-metadata.xml is changing to the Black Duck public artifactory repo, but we'll make both references honor detect.gradle.inspector.repository.url if provided.

joc0611 commented 5 years ago

Hey @stevebillings, @callmepills, and @akamen,

sorry for getting back to you so late. I have taken a look at the airgape mode, but that does not really solve my problem. I'm currently integrating hub detect into multiple Jenkins pipelines using the shared library concept. So far, I have been able to keep it build tool agnostic, the airgap mode however would require me to download Gradle Inspector beforehand, which might not be necessary, e.g. in case of an NPM project.

@stevebillings, you are right about the metadata.xml. In the case Maven Central is not accessible from within your build infrastructure, I think people will use some kind of company internal mirror of Maven Central (through tools like Nexus or Artifactory), like we do. This mirror also provides the required metadata.xml so downloading them from the same source makes sense in our use case. I agree with @callmepills in that it was surprising to see metadata.xml and actual jar files being fetched from different locations.

@stevebillings I think, this is what I implemented in pull request #353. As far as I see, this PR does not change the current behavior in the default case, but only honors detect.gradle.inspector.repository.url for both URLs, as you said. Could you have a look at this PR and tell me, if that is a viable solution in your opinion?

Thanks!

joc0611 commented 5 years ago

@stevebillings I just read

In 5.0.0 the default location of the integration-gradle-inspector jar and maven-metadata.xml is changing to the Black Duck public artifactory repo

(sorry, it was late yesterday :)) That would work for me as well!

taikuukaits commented 5 years ago

@joc0611 @callmepills

The detect air gap zip contains all of detects dependencies so it should still be build tool agnostic.

You can get the air gap zip from our public artifactory: https://test-repo.blackducksoftware.com/artifactory/webapp/#/artifacts/browse/tree/General/bds-integrations-release/com/blackducksoftware/integration/hub-detect/4.4.2/hub-detect-4.4.2-air-gap.zip

Due to the difficulty in supporting internal resources, we made the decision to draw a hard line. You either must be able to access all the public online resources detect uses or you must use air gap. In 5.0.0 we are dropping even more support of internal resources.

If you cannot use the Air Gap zip, could you please detail why?

joc0611 commented 5 years ago

@taikuukaits Technically speaking, I can use the air gap mode.

For better understanding, let me explain our infrastructure a bit: We are using an immutable build infrastructure based on Jenkins. Teams maintain their own build slave docker images, which contain the necessary build tools required for their projects to build. We have integrated hub-detect through simply calling hub-detect.sh in each projects pipeline. In case, hub-detect finds a Gradle project to scan, up to 4.3.0 it dynamically loads the Gradle Inspector from our internal repository.

To do the same in air gap mode, I think I would have to download the whole air gap zip for each run of hub-detect, right? As hub-detect requires the package managers to be available on the same machine it runs on, creating custom docker images for each team and build tool would is practically impossible for me.

Thus, using the air gap mode is technically possible, however, I preferred the idea of dynamically loading only the necessary dependencies for performance reasons.

If you have an idea how I could avoid loading the 600+MB zip for each build, I'd be grateful!

stavvy-akamen commented 5 years ago

@joc0611 could you not in theory deploy the airgap bundle on the various nodes in a known location and simply reference Detect from within that location? Obviously downloading it every time would be impractical, but the idea was to download it once. Of course if you have dozens of slave nodes and want to keep up to date it can become unwieldy, but is it at least an option?

joc0611 commented 5 years ago

@akamen I will try and find a suitable location for the bundle. That might work. I'll try it out and let you know.

On the other hand, if Gradle Inspector and the corresponding metadata.xml are both located on the blackduck artifactory in 5.0.0 as @stevebillings mentioned eralier, they are accessible from within our development infrastructure, so this would work as well.

joc0611 commented 5 years ago

@akamen But please still switch to HTTPs :)

stavvy-akamen commented 5 years ago

@joc0611 that would be great. If you can grab it from https://repo.blackducksoftware.com/artifactory/ then it is certainly ideal and using some minimal scripting, distribute the bundle across the needed nodes.

We currently do not deploy the Gradle Inspector to our AF, but that is something that @ekerwin and @taikuukaits have certainly contemplated.

callmepills commented 5 years ago

Closing the loop on my issue... I was able to get the current Maven Central location unblocked and it began working as expected. Also, I see the Gradle Inspector is already available in your AF so we should be good to go when upgraded to 5.0.0.

https://repo.blackducksoftware.com/artifactory/webapp/#/artifacts/browse/tree/General/bds-integrations-release/com/blackducksoftware/integration/integration-gradle-inspector

Thanks for the help.

taikuukaits commented 5 years ago

As of 5.0.0 you no longer need a connection to maven central, only our artifactory.

blackducksoftware / hub-detect