search

aboutcode-org / purldb

Tools to create and expose a database of purls (Package URLs). This project is sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase/ and nexB for https://www.aboutcode.org/ Chat is at https://gitter.im/aboutcode-org/discuss
https://purldb.readthedocs.io/
35 stars 23 forks source link

Enhance Alpine package scan results #307

Open aalexanderr opened 3 years ago

aalexanderr commented 3 years ago

Alpine packages lack some important info like copyrights or where the source code is located. This info can't be gathered from the packages themselves as its just not there. To get this info we need to: download aports repo & for each pkg check it out on commit specific to alpine package (via fetchcode) parse APKBUILD nexB/scancode-toolkit#2541 download package sources (fetchcode) & amend new info to package's scan results

Discussed a bit with @pombredanne Most likely @quepop will PR it

The question is- should it be standard behavior when alpine based docker is being scanned or should it be a separate pipeline?

quepop commented 3 years ago

Related: https://github.com/nexB/scancode-toolkit/issues/2541#issuecomment-856904754

quepop commented 3 years ago

@pombredanne should I PR it?

pombredanne commented 3 years ago

@quepop please do! that's much easier for review Of note:

quepop commented 3 years ago

@pombredanne I don't think there is any other way. In order to extract source variable value (or any other value) we need to run the script (APKBUILD file), because in some of the alpine packages, source variable is built in a for loop (not to mention that sometimes bash string formatting is used).

From pkg:alpine/alpine-keys@2.2-r0 APKBUILD file:

_arch_keys="
        aarch64:alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub
        armhf:alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub

        x86:alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
        x86,x86_64:alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
        x86_64:alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub

        ppc64le:alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub

        s390x:alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub

        mips64:alpine-devel@lists.alpinelinux.org-5e69ca50.rsa.pub
        "

for _i in $_arch_keys; do
        source="$source ${_i#*:}"
done
pombredanne commented 3 years ago

@quepop good point, but do we need these (e.g. these _arch_keys) ?

quepop commented 3 years ago

@pombredanne We need the source variable to be complete (to be able to download sources and scan them for copyrights). I tried to emphasize (by giving an example) that the only way we get a complete source variable every time is by running the script.

quepop commented 3 years ago

@pombredanne Regarding https://github.com/nexB/scancode-toolkit/issues/2541#issuecomment-858408580, how should I name these new files (pipe, pipeline)? My idea is /pipes/alpine_helper.py and /pipelines/alpine_complement.py

tdruez commented 3 years ago

@quepop There's already a pipes/alpine.py pipes module in which you can add the new functions. For the pipeline, what about "alpine_packages.py", unless it's too specific?

quepop commented 3 years ago

@tdruez In the case of multi-image projects we cannot match DiscoveredPackage from the database to the image (and alpine version) they are from - db entry for DiscoveredPackage only include project_id (no image_id or image_index field). Alpine version is only present in project.extra_data.images array. It's a problem because in the aports repository every alpine version is a certain branch and we need to pull them and make checkouts (branch and then commit-id) to be able to extract correct APKBUILDs.

My solution is to add a new database field to the DiscoveredPackage class named image_index which would store an index (project.extra_data.images array) of the image it is from from. Is it acceptable?

tdruez commented 3 years ago

@quepop what about adding the extra_data field on the DiscoveredPackage model, for consistency with Project and CodebaseResource models? Let me know if that would work for you and I'll make that change upstream.

pombredanne commented 3 years ago

Repasting from https://github.com/quepop/scancode.io/commit/2f59440b97cfc89e4bc8291d7839188d25e0fb34#r52902214 for reference : @quepop

I have been thinking more about this for nexB#191 and sourcing arbitrary bash script this way is too much of a security concern.

An alternative could include:

I think 2. is best and much less involved than having a container depdendency. Furthermore, there could be other variables of interest in an APKBUILD and we need eventually to parse other shell-based manifests to extract metadata such as PKGBUILD (Arch), ebuild (Gentoo) m4 (Autotools) and a few more.

Therefore I am implementing this that can then be reused here:

pombredanne commented 3 years ago

@quepop what about adding the extra_data field on the DiscoveredPackage model, for consistency with Project and CodebaseResource models? Let me know if that would work for you and I'll make that change upstream.

:+1: ... this is being added too to SCTK FWIW

quepop commented 3 years ago

@pombredanne sounds great but what about APKBUILDs like these: https://github.com/nexB/purldb/issues/307? Will it work? @tdruez Yes, it would be great.

tdruez commented 3 years ago

@quepop extra_data added on the DiscoveredPackage model https://github.com/nexB/scancode.io/pull/222 Make sure to use the DiscoveredPackage.update_extra_data() API, see https://github.com/nexB/scancode.io/blob/main/scanpipe/models.py#L197

quepop commented 3 years ago

@tdruez @pombredanne

There is another issue that i found looking at my database. Some alpine images like docker://alpine:3.13 have subpackages installed from a parent package that is not installed (libcrypto, libssl from openssl). Handling cases like these isn't clear for me (because running the scan on the parent still leaves a problem of deciding what subset of the scan's output is applicable to the parent's children) I've decided that in cases where a parent package is present, my code will update only its missing information (not its chilrdren).

pombredanne commented 3 years ago

@quepop not sure what to do wrt. subpackages... can you provide a concrete example with links and may sample APKBUILD snippets?

pombredanne commented 3 years ago

re:

@quepop not sure what to do wrt. subpackages... can you provide a concrete example with links and may sample APKBUILD snippets?

Actually the new approach in https://github.com/nexB/scancode-toolkit/blob/b7d070fc788f4c233df1f31d735e1ac5b3aa2d29/src/packagedcode/alpine.py#L144 could be enhanced to also collect these.

  1. collect the variable and resolve that lists all the subpackages such as https://gitlab.alpinelinux.org/alpine/aports/-/blob/9238f96475ab45281ce3d0cdfbf37efcbdeb8888/main/openssl/APKBUILD#L14
  2. for each of these, parse the corresponding bash function with the same name and extract the variables defined inside a back function

Then merge subpackages variables with the base variables.

This way declared licensed, description and other sub-package-specific metadata could be collected.

Note that this would work whereas the approach to just eval the APKBUILD with bash would not be able to collect variables defined inside a function, as this would require to run the function, which is impractical since the function names are not known (unless parsed ...) .

pombredanne commented 9 months ago

See https://github.com/nexB/scancode.io/pull/272#issuecomment-1964170077

The feature will be implemented instead in PurlDB ... See https://github.com/nexB/purldb/issues/95 and https://github.com/nexB/purldb/issues/237

pombredanne commented 9 months ago

Moving this to PurlDB