search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.61k stars 313 forks source link

Add BitBake (Yocto) as supported package manager #722

Open tsteenbe opened 6 years ago

tsteenbe commented 6 years ago

Add support for the BitBake (Yocto) dependency manager to the analyzer module.

Links:

BitBake manual: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html Useful Bitbake commands: https://community.nxp.com/docs/DOC-94953

Bitbake repos: https://git.yoctoproject.org/ Projects starting with poky- and meta- likely contain bitbake files

pattivacek commented 6 years ago

I recommend using the latest version of the bitbake manual: https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html. 1.6 is pretty old at this point.

sschuberth commented 5 years ago

Maybe also have a look at https://github.com/Waldleufer/archproj-bmwteam of which our contributor @andreas-bauer has a fork.

sschuberth commented 5 years ago

FYI, there's been work started in the bitbake branch.

tsteenbe commented 4 years ago

The questions below outline the prerequisites of things we need to figure out prior to be able to add Yocto support to ORT

  1. How can you detect a project uses this specific package manager?
  2. How can you get the declared license for a package?
  3. How to can one get dependency tree including package names, versions?
  4. How can one obtain the source code for a dependency? (e.g. the source code repository + revision or the sources artifact URL)
  5. How can one separate code dependencies from build/test ones?
  6. Can you provide example projects that can be used to test the implementation?
pattivacek commented 4 years ago

I think it's still an open question if it's really appropriate the use ORT on Yocto in the first place. In our case, we provide meta-updater and some additional supporting layers as open-source projects to help our users integrate aktualizr into their projects. However, meta-updater (and essentially all of Yocto) is just metadata. We don't normally deploy or deliver the software built with Yocto, just the metadata that explains how to build it.

The metadata itself that we use is released under friendly open-source terms. The software built with the metadata is more complex, and Yocto has some internal tooling for detecting it. I wrote a script that can collect the licenses for a given package and its dependencies and another script that uses the first to specifically find the dependencies of aktualizr. However, I'm not sure anymore if that is actually useful.

So before those six questions can be answered, we need to clarify: are we trying to search for licenses of the metadata that is Yocto or the underlying software that is built with Yocto?

sschuberth commented 4 years ago

are we trying to search for licenses of the metadata that is Yocto or the underlying software that is built with Yocto?

The latter, IMO. Or more generically: We need to determine the licenses of what is being distributed in the end.

pattivacek commented 4 years ago

We need to determine the licenses of what is being distributed in the end.

Even though we aren't distributing the end product ourselves? Our users normally customize their images with their own software and build them themselves. We don't deliver images as a product, but rather just the metadata to help include our software in their images.

sschuberth commented 4 years ago

Let's leave the term "(end-)product" aside for a moment. It does not matter which kind of software you distribute (e.g. whether it's a product, SDK, or some tools that build the actual "product"), but if you distribute software (esp. in binary form), you need to take care to fulfill all obligations of the licenses of the software you are distributing.

So if you as a company ship a .tar.gz archive to your customers that contains software (or metadata) that allows your customers to build their own binaries of something, then you still need to gather the licenses and assemble a disclosure document for the software (or metadata) in the .tar.gz archive. I hope that makes it clearer.

mmurto commented 4 years ago

5. How can one separate code dependencies from build/test ones?

We're currently trying to approach this with file level, rather than package level license granularity, so it might not be completely relevant for ORT, but here's where we're at:

Yocto has some built in tooling to scan the built image for source files used to build it. We're using this to get the hash values to find licenses of individual files, but it the tooling might be useful for separating the code in the build from build/test ones for you aswell. We've described the process in a little more detail here (heavily WIP).

pattivacek commented 4 years ago

It does not matter which kind of software you distribute (e.g. whether it's a product, SDK, or some tools that build the actual "product"), but if you distribute software (esp. in binary form), you need to take care to fulfill all obligations of the licenses of the software you are distributing.

Exactly. We only distribute one project's source code (which is C++, so we can scan it directly instead of going through Yocto) and the rest is just metadata. We can scan the metadata, but in our case, I don't believe it makes sense to scan the software described by the metadata. We don't distribute that software, just (some of) the metadata.

That said, that's just our use case. I don't know if there are others using Yocto to build images that then get distributed, but we (the Edge OTA team) do not do that.

sschuberth commented 6 months ago

For reference, I just learned that besides Double Open's original https://github.com/doubleopen-project/meta-doubleopen, there's also https://github.com/fossas/meta-fossa that might be worth looking at.

tsteenbe commented 6 months ago

Also https://github.com/fosslight/fosslight_yocto_scanner may be worth looking into..