Execution failed for task ':snykResolvedDepsJson'.

candrews commented 4 years ago

node -v: v12.16.3
npm -v: 6.14.4
snyk -v: 1.341.1
Command run: snyk test

Expected behaviour

Dependencies analyzed

Actual behaviour

Gradle Error (short):
> Could not resolve all dependencies for configuration ':snykMergedDepsConf'.
   > Could not resolve org.slf4j:slf4j-simple:1.8.0-beta4.
      > Module 'org.slf4j:slf4j-simple' has been rejected:
   > Could not resolve org.slf4j:slf4j-simple:1.7.30.
      > Module 'org.slf4j:slf4j-simple' has been rejected:
   > Could not resolve ch.qos.logback:logback-classic:1.2.3.
      > Module 'ch.qos.logback:logback-classic' has been rejected:
   > Could not resolve ch.qos.logback:logback-classic.
      > Module 'ch.qos.logback:logback-classic' has been rejected:

===== DEBUG INFORMATION START =====
gradle command: '/home/candrews/Downloads/demo/gradlew' snykResolvedDepsJson -q --build-file build.gradle --no-daemon -Dorg.gradle.parallel= -Dorg.gradle.console=plain -PonlySubProject=. -I /tmp/tmp-49189-hgINAGkBbrXw--init.gradle

------------------------------------------------------------
Gradle 6.4.1
------------------------------------------------------------

Build time:   2020-05-15 19:43:40 UTC
Revision:     1a04183c502614b5c80e33d603074e0b4a2777c5

Kotlin:       1.3.71
Groovy:       2.5.10
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          11.0.7 (Oracle Corporation 11.0.7+10)
OS:           Linux 5.6.18-300.fc32.x86_64 amd64

>>> command: '/home/candrews/Downloads/demo/gradlew' snykResolvedDepsJson -q --build-file build.gradle --no-daemon -Dorg.gradle.parallel= -Dorg.gradle.console=plain -PonlySubProject=. -I /tmp/tmp-49189-hgINAGkBbrXw--init.gradle
>>> exit code: 1
>>> stdout:
SNYKECHO snykResolvedDepsJson task is executing via doLast
JSONATTRS {"org.gradle.usage":["java-runtime","java-api"],"org.gradle.category":["library"],"org.gradle.libraryelements":["jar"],"org.gradle.dependency.bundling":["external"]}
SNYKECHO processing project: demo
SNYKECHO constructing merged configuration from [allResolvable, annotationProcessor, apiElements, archives, bootArchives, compile, compileClasspath, compileOnly, default, developmentOnly, implementation, productionRuntimeClasspath, runtime, runtimeClasspath, runtimeElements, runtimeOnly, sonarlint, sonarlintPlugins, spotbugs, spotbugsPlugins, spotbugsSlf4j, testAnnotationProcessor, testCompile, testCompileClasspath, testCompileOnly, testImplementation, testRuntime, testRuntimeClasspath, testRuntimeOnly]
SNYKECHO resolving configuration snykMergedDepsConf

>>> stderr:

FAILURE: Build failed with an exception.

* Where:
Initialization script '/tmp/tmp-49189-hgINAGkBbrXw--init.gradle' line: 260

* What went wrong:
Execution failed for task ':snykResolvedDepsJson'.
> Could not resolve all dependencies for configuration ':snykMergedDepsConf'.
   > Could not resolve org.slf4j:slf4j-simple:1.8.0-beta4.
     Required by:
         project :
      > Module 'org.slf4j:slf4j-simple' has been rejected:
           Cannot select module with conflict on capability 'logging:slf4j-impl-capability:0' also provided by [ch.qos.logback:logback-classic:1.2.3(runtime)]
   > Could not resolve org.slf4j:slf4j-simple:1.7.30.
     Required by:
         project : > org.springframework.boot:spring-boot-starter:2.3.1.RELEASE > org.springframework.boot:spring-boot-dependencies:2.3.1.RELEASE
      > Module 'org.slf4j:slf4j-simple' has been rejected:
           Cannot select module with conflict on capability 'logging:slf4j-impl-capability:0' also provided by [ch.qos.logback:logback-classic:1.2.3(runtime)]
   > Could not resolve ch.qos.logback:logback-classic:1.2.3.
     Required by:
         project : > org.springframework.boot:spring-boot-starter:2.3.1.RELEASE > org.springframework.boot:spring-boot-dependencies:2.3.1.RELEASE
      > Module 'ch.qos.logback:logback-classic' has been rejected:
           Cannot select module with conflict on capability 'logging:slf4j-impl-capability:0' also provided by [org.slf4j:slf4j-simple:1.8.0-beta4(runtime)]
   > Could not resolve ch.qos.logback:logback-classic.
     Required by:
         project : > org.springframework.boot:spring-boot-starter:2.3.1.RELEASE > org.springframework.boot:spring-boot-starter-logging:2.3.1.RELEASE
      > Module 'ch.qos.logback:logback-classic' has been rejected:
           Cannot select module with conflict on capability 'logging:slf4j-impl-capability:0' also provided by [org.slf4j:slf4j-simple:1.8.0-beta4(runtime)]

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 10s

===== DEBUG INFORMATION END =====

Error running Gradle dependency analysis.

Please ensure you are calling the `snyk` command with correct arguments.
If the problem persists, contact support@snyk.io, providing the full error
message from above, starting with ===== DEBUG INFORMATION START =====.

[2]

Steps to reproduce

Use this build.gradle
run snyk test

If applicable, please append the --debug flag on your command and include the output here **ensuring to remove any sensitive/personal details or tokens.

anthogez commented 4 years ago

Hi @candrews thanks for raising this issue!

I would like to say that this issue might not be related to our tool but due of a detection of incompatible dependencies in your project

meanwhile I would like to recommend you this article about this problem https://docs.gradle.org/current/userguide/component_capabilities.html

If even after the use of outgoing capability rules, this problem persists contact us again. Thanks!

candrews commented 4 years ago

If I run gradle build it completes successfully.

The problem only occurs when running snyk test.

If it was a problem with gradle, another plugin, or a dependency, gradle build would also fail.

shanman190 commented 4 years ago

@anthogez and @candrews,

This is as a result of the new component metadata rules that you, @anthogez, linked. This manifests because of the snykConf includes all of the other configurations in the model. As a result logback-classic (implementation conf) and slf4j-simple (testImplementation conf) are present same configuration with the same capability declared when component metadata rules are evaluated.

@candrews, a workaround would be to use the --configuration-matching flag with the configuration that you want to test. That should allow you to get past this issue in the short term.

lili2311 commented 4 years ago

@candrews 👋 Could you tell us a little bit more about your project please? We are looking at improving this behaviour.

is it an Andoroid project with build flavours at all?
gradle build is the only command you run, can you tell us more about the setup and what is executed here by default? How was it configured/setup? Is is a task/buildscript? If your project is open source please do share a link so we can try this all locally. Or is the only file at all the build.gradle you shared?

You can see and target some configuration only by running gradle properties to see what is being used, I can suggest scanning with --configuration-matching=^productionReleaseRuntimeClasspath$|^releaseRuntimeClasspath$|^runtimeClasspath$ or implementation depending on what configurations you have

candrews commented 4 years ago

* is it an Andoroid project with build flavours at all?

No, it's not Android. It's a Java on Spring Boot project.

* `gradle build` is the only command you run, can you tell us more about the setup and what is executed here by default? How was it configured/setup? Is is a task/buildscript? If your project is open source please do share a link so we can try this all locally. Or is the only file at all the `build.gradle` you shared?

The project is not open source, unfortunately, but I did make a reproducible test case that is. I made the test case starting with https://start.spring.io/ then adding additional gradle plugins. Here's the full project: demo.zip

You can see and target some configuration only by running gradle properties to see what is being used, I can suggest scanning with --configuration-matching=^productionReleaseRuntimeClasspath$|^releaseRuntimeClasspath$|^runtimeClasspath$ or implementation depending on what configurations you have

My organization mandates the use of snyk and they dictate the configuration of it, including command line options. So unfortunately, such a workaround won't help me. I have my :crossed_fingers: that snyk itself can be fixed so my organization can upgrade to a fixed version eliminating this issue.

Thank you again!

shanman190 commented 4 years ago

@lili2311,

Not sure how familiar you all are with some of the newer features, but I'll explain a little bit in case it's not very familiar.

With Gradle 5+, Gradle introduces component capabilities and module metadata. Capabilities are used to describe what features a library provides from a build engine standpoint and module metadata provides a way for Gradle to publish that a library provides the declared set of capabilities defined by a library's author(s). Throughout Gradle 5.x module metadata was strictly an opt-in feature, but with Gradle 6+ it is now enabled by default. From 5.x onward, if a dependency is consumed that contains a module metadata descriptor, Gradle will enhance the model with the declared capabilities from that descriptor. Once Gradle has those declared capabilities to work with, it can throw an error when there are two libraries that provide the same capability in that a given configuration as it would be aware enough to know that there would be a conflict. In this particular case both logback-classic and slf4j-simple expose the same logging capability and thus produce an error. Essentially on a JVM based application, there should only ever be a single logging implementation binding/provider on the classpath at a given time and having multiple present in a lot of cases lead to not getting any logs at all.

It is similar to Android build variants, except being more generically defined, so that standard JVM library and application authors can also take advantage of the similar feature set.

candrews commented 4 years ago

If snyk would like me to test a workaround using capabilities, I'd be happy to do if provided with the necessary additions to build.gradle.

Are we in agreement that this is a problem with the snyk gradle plugin and that the team is working on fixing it?

shanman190 commented 4 years ago

@candrews, just some quick double checking but both slf4j and logback are built using Maven, so Gradle module metadata wouldn't be published natively for those modules. However, what that tells me is that your buildscript is enhancing the model via some configuration or a plugin. Assuming that you can drop that, which understandably isn't ideal, that should allow the snyk test to complete successfully while also satisfying your conditions for not being able to control the cli options. Doing so would also help to confirm the suspicion about the issue being tied directly to Gradle capabilities and not something else going on in your build.

anthogez commented 4 years ago

Hi @candrews, I would like to say and make clear that yes, we are investigating the issue and we are doing our best to unlock you. Thanks for your patience

shanman190 commented 4 years ago

Here's a mostly full reproducible, you just need to add the two additional Gradle plugins found in the original build.gradle script, sample from https://start.spring.io: here

@candrews, did some testing by using the above sample. With that sample as is, I was able to run snyk test with no modifications. At that point, I began to layer on your extra plugins to see which one was adding the capabilities. In this specific case, it's the name.remal.sonarlint that is the culprit that introduces these. If you remove it completely, the project can be scanned without any issues or alternatively, disable the name.remal.component-capabilities sub-plugin via your gradle.properties file also appears to be a potential workaround.

plugin-disabled.name.remal.component-capabilities=true

Source for the name.remal.* plugins: https://gitlab.com/remal/gradle-plugins/

lili2311 commented 4 years ago

👋 @candrews you can scan this by running snyk test --configuration-matching="^.*untime.*$|^.*implementation.*$" As advised previously you have to target some configurations to scan when there is a clash, this also helps filter out test dependencies and others that are not production level.

To get to this I looked at the DEBUG=*snyk* snyk test output where all the available configurations are listed:

  SNYKECHO constructing merged configuration from [allResolvable, annotationProcessor, api, apiElements, archives, bom, bootArchives, checkstyle, compile, compileClasspath, compileOnly, default, developmentOnly, implementation, jacocoAgent, jacocoAnt, javadocElements, pmd, productionRuntimeClasspath, runtime, runtimeClasspath, runtimeElements, runtimeOnly, sonarlint, sonarlintPlugins, sourcesElements, spotbugs, spotbugsPlugins, spotbugsSlf4j, testAnnotationProcessor, testCompile, testCompileClasspath, testCompileOnly, testImplementation, testRuntime, testRuntimeClasspath, testRuntimeOnly]

the same can be achieved by looking in gradle dependencies -q, this list then can be used to pick which configurations are production like and construct the pattern. I was able to grab 48 dependencies with the command provided:

We are looking at automating/improving this experience to match closely what gradle build picks up instead, for now the provided example should help you get the scan results.

lili2311 commented 4 years ago

👋 @candrews did this work for you?

candrews commented 4 years ago

plugin-disabled.name.remal.component-capabilities=true

That worked around the issue successfully.

We are looking at automating/improving this experience to match closely what gradle build picks up instead, for now the provided example should help you get the scan results.

Is there an issue I could follow to learn when this work is in place (or perhaps this is the issue to follow)?

Thank you again!

lili2311 commented 4 years ago

Great to hear this is working for you, we can keep this issue open for further updates on the plans forward.

remal commented 4 years ago

Hello Snyk Gradle plugin authors,

I'm the author of name.remal.component-capabilities Gradle plugin.

The suggested solution here (plugin-disabled.name.remal.component-capabilities=true) definitely helps (and, BTW, I've already created a fix that makes it redundant), but...

I would say that it's not the best idea to merge all dependencies in one configuration, as Gradle dependency resolution is quite complex a lot of issues (like this) can happen.

I tried to research if it's possible to define component capabilities only for compile/runtime classpath configurations and couldn't find a way to do it.

As component capabilities can be defined by any plugin or developer himself, I would say that it makes sense to change Snyk behaviour from working with "merged" dependencies to Gradle configurations separately.

lili2311 commented 4 years ago

Snyk does offer ability to target configurations separately (this is not popular to use per configuration as you will end up with lots of "projects" in snyk), but it seems desirable by the users to scan all dependencies that would be brought in during a build or even for test to get the full overview of all dependencies in application and remediate any that have vulnerabilities. We will continue looking at ways to plugin in to gradles ability to tell us this and see if we can find a way to make this easy and similar to whatever build command does under the hood.

remal commented 3 years ago

@anthogez As I can see, dependencies resolution logic was changed on Nov 29, 2020. Seems init.gradle doesn't merge all Gradle configurations into snykMergedDepsConf anymore and works with configurations separately. So... Hasn't the issue been resolved?

anthogez commented 3 years ago

Hey @remal, thanks for reaching out - yes this issue has been resolved we are closing this one 😃 thanks!

snyk / snyk-gradle-plugin