search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.53k stars 300 forks source link

Take a look at Google's Insights project (https://deps.dev/) #4112

Open sschuberth opened 3 years ago

sschuberth commented 3 years ago

Google just announced the Insights project. In large parts it has similarities with ORT's analyzer and advisor components, and pages like this one have similarities with the web app report.

I'll reach out to the Insights team to learn about how they are doing things.

sschuberth commented 3 years ago

Their FAQ is a very interesting read:

"How is Insights implemented?

Most of the back end is written in Go and reimplements the dependency analysis algorithm (version resolution) of each packaging system. The code has been tested for agreement with the native tooling, but is faster because it is decoupled from package installation. It constructs the graph without actually building and installing the software."

tsteenbe commented 3 years ago

Sofar insights projects only npm, Maven, Go modules, and Cargo for which static analysis should work really well (apart from maybe Go). Would be interested to learn how they did Go support and how they are planning to do PyPI.

sschuberth commented 3 years ago

I'll share what can be learned from their response once I get it.

sschuberth commented 2 years ago

Insight's data is now also available via BigQuery. Maybe this data could be used by the Analyzer to speed up the lookup of package metadata.

sschuberth commented 2 years ago

I'll share what can be learned from their response once I get it.

BTW, I never heard back from them, so I just reached out again.

sschuberth commented 4 months ago

BTW, the deps.dev people posted a great blog article that highlights the importance of "context" during dependency resolution, but their API currently doesn't offer a way to provide this context.

Aponace commented 4 months ago

@sschuberth I have actually implemented an identical mechanism about a year ago in Python and Neo4J. From my experience, aside from it being cool and all, you need a very large infrastructure that's being updated with new version releases (which even deps.dev doesn't do - they update maybe twice a year) and a complex resolution logic to match the actual tree to the one generated by the package managers (which depends on the context of where the package is used and alongside which other packages). In other words, it's not a practical solution unless you have a lot of money and research invested in it on a dedicated central infrastructure. You can't get the exact picture of the dependency tree unless you use the native package managers resolution mechanism.

sschuberth commented 4 months ago

You can't get the exact picture of the dependency tree unless you use the native package managers resolution mechanism.

I absolutely agree @Aponace, and that's why ORT's analyzer has always aimed to use the native package manager's resolution mechanism.

However, it was brought to my attention that several ORT users are not necessarily interested in the exact picture, but something "close enough" (that can be determined much faster) is sufficient. From a license compliance aspect, that might even be true, as licenses usually do not change between "similar" package versions. But from a security compliance aspect, that might be a problem, as also patch level package versions may introduce severe security risks.

sschuberth commented 3 months ago

Their new API now also supports PURLs.

sschuberth commented 3 months ago

Also @jamietanna's Dependency Management Data (DMD) seems to be using deps.dev, which might be an interesting integration point.

jamietanna commented 3 months ago

Thanks Sebastian, yes, you should be able to use DMD alongside ORT (https://dmd.tanna.dev/cookbooks/getting-started-ort/) in the meantime to get this data! But would be great to have first-class support

heliocastro commented 3 months ago

@jamietanna Just to mention, the docs are outdated, we moved from ort-extended to ort a while ago. We have this two images: ghcr.io/oss-review-toolkit/ort-minimal - The (kind of) minimal image with python, rust, ruby and node ghcr.io/oss-review-toolkit/ort - The one that have everything and is the direct replacement for ort-extended

jamietanna commented 3 months ago

@jamietanna Just to mention, the docs are outdated, we moved from ort-extended to ort a while ago. We have this two images: ghcr.io/oss-review-toolkit/ort-minimal - The (kind of) minimal image with python, rust, ruby and node ghcr.io/oss-review-toolkit/ort - The one that have everything and is the direct replacement for ort-extended

Thanks! I'm on holiday for the next week, but can look to update when I'm home. Or I'd welcome a Merge Request on GitLab if you were happy to update for me ☺