Generate dependency tree in dot format

shanman190 commented 4 years ago

With very large projects, this initscript has some problems with regards to memory usage since it is attempting to keep the entire dependency tree in memory through any included configurations or attributes. In my Gradle version at least it also appeared that the snykMergedDepsConf would result in a cycle, but dependency cycles in Gradle are not allowed within a configuration, so the combination of all configurations should also not have any cycles.

[ ] Ready for review
[ ] Follows CONTRIBUTING rules
[ ] Reviewed by Snyk internal team

What does this PR do?

Converts the graph generation to use dot format. This alleviates the memory consumption problem for large projects as well as speeding up graph generation.

Where should the reviewer start?

Most of the code is in the init.gradle file. However, the parse-gradle.ts was scavenged pretty close to verbatim from the snyk-mvn-plugin.

How should this be manually tested?

I wasn't sure how to run the plugin through the CLI directly, so I was mostly running this directly via the Gradle commands:

./gradlew -I lib/init.gradle snykResolvedDeps -q --no-daemon -Dorg.gradle.parallel=
./gradlew -I lib/init.gradle snykResolvedDeps -q --no-daemon -Dorg.gradle.parallel= -Pconfiguration=testRuntime
./gradlew -I lib/init.gradle snykResolvedDeps -q --no-daemon -Dorg.gradle.parallel= -PonlySubProject=sub-project

All commands should remain identical to their previous counterparts.

What are the relevant tickets?

Snyk support ticket 3464

claassistantio commented 4 years ago

All committers have signed the CLA.

shanman190 commented 4 years ago

So I'm not very fluent in Typescript. I know enough to be able to read it and hack my way through it. However, this implementation specifically, isn't panning out the way that I was hoping that it would.

I realize that with the original implementation it is technically the most accurate as it actually represented the dependency graph to it's fullest extent (as a graph via recursion through the dependencies). In my particular case, I've got a project with a few hundred dependencies and paths to several of those occur numerous times in my graph. As a result, memory spins out of control with this implementation. It was my hope that I would be able to come up with a solution that was both memory efficient as well as keeping to the complete picture attribute.

I'll be returning to the drawing board and see if I can't come up with something that stays more on the Gradle side of things.

shanman190 commented 4 years ago

One question that does come to mind is if Snyk is actually utilizing the Gradle configuration being written out in the json. Looking through the UI, it doesn't appear to be represented anywhere, but I also may not be looking in the right place.

lili2311 commented 4 years ago

👋 @shanman190, thank you very much for taking the time to do this. This is indeed something we would like to tackle, would you be willing to pair with us on this?

We would like to have a chat / zoom to run thought your proposal and see if it fits our product constrains, we can then pair on the solution as we can guide you with TypeScript where needed and with product decisions :)

shanman190 commented 4 years ago

👋 @lili2311, yes absolutely! How best would it be to get something like that scheduled with you all? (PS. I'm in US Central Time zone, since I noticed you appear to be based out of London)

orsagie commented 4 years ago

👋 @shanman190, we schedule a time around your early afternoon and our late evening. Can you send your email to or@snyk.io so we can figure out a good day?

aviadhahami commented 4 years ago

Hey @shanman190 ! Thanks for talking w/ @orsagie and myself last night :man_technologist: :woman_technologist:

We'll merge your branch soon, you can follow the progress here https://github.com/snyk/snyk-gradle-plugin/pull/113 :eyes:

Thanks again! :+1:

snyk / snyk-gradle-plugin