Closed joc0611 closed 5 years ago
@joc0611 Interesting, so you are reporting a regression then. We shall investigate.
Hi Johannes: After considering several challenges related to "partially online" environments (where some sites are accessible, but others are not), we've concluded that the least-confusing approach overall is to require that detect either run fully online (with access to all sites it uses), or run in airgap mode.
This case is a good example. If we make the change you're suggesting, it does solve the problem, but it would require that the user deploy not just the jar, but also the maven-metadata.xml file. And detect would then stop upgrading to the latest version of the gradle inspector even if configured to do so, a fact which may come as a surprise to some users.
So, we recommend running in airgap mode. Would that work for you?
I am facing the same issue now in my environment. The default, hard coded metadata URL is currently blocked:
However, the following URL is unblocked for me:
Should we switch to this?
Either way, it does feel weird that the JAR and metadata could be resolved/downloaded from two different repositories. If present, I think the custom repository URL should override both.
ah, I see. You have a good point. We are planning to resolve this in detect 5.0.0. (In 5.0.0 the default location of the integration-gradle-inspector jar and maven-metadata.xml is changing to the Black Duck public artifactory repo, but we'll make both references honor detect.gradle.inspector.repository.url if provided.
Hey @stevebillings, @callmepills, and @akamen,
sorry for getting back to you so late. I have taken a look at the airgape mode, but that does not really solve my problem. I'm currently integrating hub detect into multiple Jenkins pipelines using the shared library concept. So far, I have been able to keep it build tool agnostic, the airgap mode however would require me to download Gradle Inspector beforehand, which might not be necessary, e.g. in case of an NPM project.
@stevebillings, you are right about the metadata.xml
. In the case Maven Central is not accessible from within your build infrastructure, I think people will use some kind of company internal mirror of Maven Central (through tools like Nexus or Artifactory), like we do. This mirror also provides the required metadata.xml
so downloading them from the same source makes sense in our use case. I agree with @callmepills in that it was surprising to see metadata.xml
and actual jar
files being fetched from different locations.
@stevebillings I think, this is what I implemented in pull request #353. As far as I see, this PR does not change the current behavior in the default case, but only honors detect.gradle.inspector.repository.url
for both URLs, as you said. Could you have a look at this PR and tell me, if that is a viable solution in your opinion?
Thanks!
@stevebillings I just read
In 5.0.0 the default location of the integration-gradle-inspector jar and maven-metadata.xml is changing to the Black Duck public artifactory repo
(sorry, it was late yesterday :)) That would work for me as well!
@joc0611 @callmepills
The detect air gap zip contains all of detects dependencies so it should still be build tool agnostic.
You can get the air gap zip from our public artifactory: https://test-repo.blackducksoftware.com/artifactory/webapp/#/artifacts/browse/tree/General/bds-integrations-release/com/blackducksoftware/integration/hub-detect/4.4.2/hub-detect-4.4.2-air-gap.zip
Due to the difficulty in supporting internal resources, we made the decision to draw a hard line. You either must be able to access all the public online resources detect uses or you must use air gap. In 5.0.0 we are dropping even more support of internal resources.
If you cannot use the Air Gap zip, could you please detail why?
@taikuukaits Technically speaking, I can use the air gap mode.
For better understanding, let me explain our infrastructure a bit: We are using an immutable build infrastructure based on Jenkins. Teams maintain their own build slave docker images, which contain the necessary build tools required for their projects to build. We have integrated hub-detect through simply calling hub-detect.sh
in each projects pipeline. In case, hub-detect finds a Gradle project to scan, up to 4.3.0 it dynamically loads the Gradle Inspector from our internal repository.
To do the same in air gap mode, I think I would have to download the whole air gap zip for each run of hub-detect, right? As hub-detect requires the package managers to be available on the same machine it runs on, creating custom docker images for each team and build tool would is practically impossible for me.
Thus, using the air gap mode is technically possible, however, I preferred the idea of dynamically loading only the necessary dependencies for performance reasons.
If you have an idea how I could avoid loading the 600+MB zip for each build, I'd be grateful!
@joc0611 could you not in theory deploy the airgap bundle on the various nodes in a known location and simply reference Detect from within that location? Obviously downloading it every time would be impractical, but the idea was to download it once. Of course if you have dozens of slave nodes and want to keep up to date it can become unwieldy, but is it at least an option?
@akamen I will try and find a suitable location for the bundle. That might work. I'll try it out and let you know.
On the other hand, if Gradle Inspector and the corresponding metadata.xml
are both located on the blackduck artifactory in 5.0.0 as @stevebillings mentioned eralier, they are accessible from within our development infrastructure, so this would work as well.
@akamen But please still switch to HTTPs :)
@joc0611 that would be great. If you can grab it from https://repo.blackducksoftware.com/artifactory/ then it is certainly ideal and using some minimal scripting, distribute the bundle across the needed nodes.
We currently do not deploy the Gradle Inspector to our AF, but that is something that @ekerwin and @taikuukaits have certainly contemplated.
Closing the loop on my issue... I was able to get the current Maven Central location unblocked and it began working as expected. Also, I see the Gradle Inspector is already available in your AF so we should be good to go when upgraded to 5.0.0.
Thanks for the help.
As of 5.0.0 you no longer need a connection to maven central, only our artifactory.
Loading Gradle Inspector Requires Connection to Maven Central
Expected behavior
When setting a repository URL and fixed version without wildcards for the Gradle inspector (through
detect.gradle.inspector.repository.url
anddetect.gradle.inspector.version
, no connection to Maven Central (http://repo2.maven.org) is established.Actual behavior
Even though
detect.gradle.inspector.repository.url=https://my-maven-central.url/
anddetect.gradle.inspector.version=0.7.0
is set, a connection to Maven Central (http://repo2.maven.org) is established.Steps to Reproduce
Version
hub-detect version: 4.3.1 and higher
Problem description
There might be build environments, where access to Maven Central is not possible due to corporate policy. In such cases, it should be possible to download the Gradle Inspector from a mirror repository.
This was possible in hub-detect version 4.3.0 by setting the parameters
detect.gradle.inspector.repository.url
anddetect.gradle.inspector.version
as shown above.In more recent versions, a connection to Maven Central is established in any case, causing an exception to be thrown, which eventually fails the execution of hub detect.
I think the relevant commit is ccdac43fa8297cdf7b82eb3977b682316181bd6f
Possible solutions
I think there are two possible solutions:
metadata.xml
in case it is not a range (this is the behavior from 4.3.0)detect.gradle.inspector.repository.url
to do the version resolutionMisc
Also, I think it would be great to make the hard-coded URL configurable and use an HTTPS connection in the default value.