cirocosta commented 5 years ago

generation of bill of materials (bom)

Having most of the "source to container image" functionality figured out, now we have to get better at generating the bill of materials.

Here are some steps to make that better:

[x] refactor packages
- [x] get away from apt install
- retrieve deb uris (apt-get --print-uris $LIST_OF_PACKAGES)
- gather the packages (wget -i $uris_file) and inspect their control file (for deb in *.deb; do dpkg-deb -I $deb control)
  - see https://www.debian.org/doc/debian-policy/ch-controlfields.html#syntax-of-control-files
- install with dpkg
  - dpkg -i *.deb
- for each package, verify if source packages can be found for that version
  - for pkg in $list_of_packages; do apt-get source --print-uris $pkg ; done (so that it can fail properly)
  - take note of packages without sources
- produce BOM
[x] refactor file addition
- compute the file digest (estaleiro digest --filename=<> --algorithm=sha256sum)
- when copying, copy both file (to regular dest) and digest (to /var/lib/estaleiro/something.digest)
[x] metadata injection
- add a final bom.yml to the labelset
[x] refactor step addition
- should provide a digest too

Below, some context.

the problem space

For each source components that bring artifacts into the final image, we must be able to keep record of such additions.

At the moment, there are three ways of getting stuff in:

base image
files from steps
files from tarballs
debian packages

While the first three have no "build-time" dynamic aspect to them, the last one does - a package might have many other dependencies that we don't know beforehand.

That means that we have two types of BOM generation:

static
dynamic

For 1, we can populate the BOM struct from within the LLB generation code.

For 2, we must do it from steps at runtime, persisting their results into files that can be accessed later.

format

base image

image "blabla" {
  base_image {
    name = "this"
    ref  = "that"
  }
}

produces

base_image:
  name: "docker.io/library/this"
  digest: "sha256@ahushaui"
  packages:
    - name: vim-runtime
      version: 1.29b-2ubuntu0.1
      source_package: vim

packages

image "bla" {
  package "this" {}
}

produces

changeset:
  packages:
    - name: zstd
      version: '1.3.3+dfsg-2ubuntu1'
      url: http://archive.ubuntu.com/ubuntu/pool/universe/libz/libzstd/zstd_1.3.3+dfsg-2ubuntu1_amd64.deb
      digest: 'SHA256:c85b2abcddbd7abc07fb06bc3a1b3fb6b80c2316e787abe05bb4d6909dc831f2'
      source_package: higher-level-package
      source:
        - url: http://archive.ubuntu.com/ubuntu/pool/main/libz/libzstd/libzstd_1.3.3+dfsg-2ubuntu1.dsc
          name: libzstd_1.3.3+dfsg-2ubuntu1.dsc
          type: deb-src
          digest: SHA256:c28c88103e3b8eecd5361bf38b185d1ac4a02712e153786ea4d01d26fea6eeb0

files

from steps

image "blabla" {
  file "/usr/local/bin/estaleiro" {
    from_step "estaleiro" {
      path = "/usr/bin/estaleiro"
    }
  }
}

step "estaleiro" {
  dockerfile = "./Dockerfile"
  target = "build"

  source_file "/usr/bin/estaleiro" {
    vcs {
      repository = "https://github.com/cirocosta/estaleiro"
      ref        = "master"
    }
  }
}

produces

changeset:
  files:
    - name: "/usr/local/concourse/bin/concourse"
      digest: "sha256:huidashiu"
      from_step:
        name: "estaleiro"
      source:
        - url: https://github.com/concourse/concourse
          type: git
          ref: master

from tarballs

image "blabla" {
  file "/usr/local/concourse/bin/concourse" {
    from_tarball "linux-rc" {
      path = "/concourse/bin/concourse"
    }
  }
}

tarball "linux-rc" {
  source_file "concourse/bin/concourse" {
    vcs {
      repository = "https://github.com/concourse/concourse"
      ref        = "master"
    }
  }
}

produces

changeset:
  files:
    - name: "/usr/local/concourse/bin/concourse"
      digest: "sha256:huidashiu"
      source:
        - url: https://github.com/concourse/concourse
          type: git
          ref: master
      from_tarball:
        name: "linux-rc"
        digest: "sha256:ahuhsui"

cirocosta commented 5 years ago

🙌

- name: vim-runtime
  version: 2:8.0.1453-1ubuntu1.1
  source_package: vim
  location:
      uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim-runtime_8.0.1453-1ubuntu1.1_all.deb
      name: vim-runtime_2%3a8.0.1453-1ubuntu1.1_all.deb
      size: "5435124"
      digest: MD5Sum:0ec3148ec2668dfbb8d357dc8f6240e6
  source:
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1.dsc
      name: vim_8.0.1453-1ubuntu1.1.dsc
      size: "2934"
      digest: SHA256:c85b2abcddbd7abc07fb06bc3a1b3fb6b80c2316e787abe05bb4d6909dc831f2
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453.orig.tar.gz
      name: vim_8.0.1453.orig.tar.gz
      size: "13434095"
      digest: SHA256:ddf3f1baf0aa8f2a988bd6ef3ee305a6cd99f365de9024faa2827a1344be8679
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1.debian.tar.xz
      name: vim_8.0.1453-1ubuntu1.1.debian.tar.xz
      size: "190292"
      digest: SHA256:97553c5f79470dba084e5de2e33805c1222e5233c1d5fb31866fd5bf90d611ec
- name: vim
  version: 2:8.0.1453-1ubuntu1.1
  source_package: ""
  location:
      uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1_amd64.deb
      name: vim_2%3a8.0.1453-1ubuntu1.1_amd64.deb
      size: "1151556"
      digest: MD5Sum:a1fc106a6538bb091ef9787c07c69de0
  source:
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1.dsc
      name: vim_8.0.1453-1ubuntu1.1.dsc
      size: "2934"
      digest: SHA256:c85b2abcddbd7abc07fb06bc3a1b3fb6b80c2316e787abe05bb4d6909dc831f2
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453.orig.tar.gz
      name: vim_8.0.1453.orig.tar.gz
      size: "13434095"
      digest: SHA256:ddf3f1baf0aa8f2a988bd6ef3ee305a6cd99f365de9024faa2827a1344be8679
    - uri: http://archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1.debian.tar.xz
      name: vim_8.0.1453-1ubuntu1.1.debian.tar.xz
      size: "190292"
      digest: SHA256:97553c5f79470dba084e5de2e33805c1222e5233c1d5fb31866fd5bf90d611ec

cirocosta commented 5 years ago

damn, it turns out that you can't just drop dpkg -i *.deb and expect it to work for multiple packages 😅

e.g., estaleiro apt vim build-essential btrfs-tools altogether --> failure :(

going to have to refactor that part ...

cirocosta commented 5 years ago

as I'm definitely not in the business of solving deb dependency trees, it seems like the best approach is to provide all of the debian packages through a local trusted directory (deb [trusted=yes] file:/var/lib/estaleiro/apt ./) so that after we got all of the information necessary from all of the upstream repositories we can just remove everything from sources.list and then go on with what we got locally, preventing any surprises in terms of reaching to the network to retrieve extra packages (with potentially different versions).

cirocosta commented 5 years ago

one caveat is that it's necessary to generate a Package file that lists all packages, and what seems the best way of doing so is leveraging dpkg-dev (which has many dependencies 😅 )

I'm thinking that it might be good to do all of this outside of the tree of fs changes that leads to the final image in order to avoid polluting it

cirocosta commented 5 years ago

we could also collect os-release and make it part of the final bom 🤔

cirocosta commented 5 years ago

Moving towards a place where we don't store those bill of materials files along the way in each layer that estaleiro runs on, we could perhaps have those coming from a straight transformation just in the final layer right before the last snapshot 🤔

cirocosta commented 5 years ago

kind: packages/v1
data:
    initial: false
    packages:
      - name: libssl1.1
        version: 1.1.1-1ubuntu2.1~18.04.4
        source_package: openssl
        architecture: amd64
        location:
            uri: http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1-1ubuntu2.1~18.04.4_amd64.deb
            name: libssl1.1_1.1.1-1ubuntu2.1~18.04.4_amd64.deb
            size: "1299616"
            md5sum: 41f3ea2b9f5b419550f165975b941f81
        source:
          - uri: http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1-1ubuntu2.1~18.04.4.dsc
            name: openssl_1.1.1-1ubuntu2.1~18.04.4.dsc
            size: "2776"
            md5sum: 88218150efac41c72aaf0025cd4d481800e0871e5ea045d25c5b10b09f7b0a88
          - uri: http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1.orig.tar.gz
            name: openssl_1.1.1.orig.tar.gz
            size: "8337920"
            md5sum: 2836875a0f89c03d0fdf483941512613a50cfb421d6fd94b9f41d7279d586a3d
          - uri: http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1.orig.tar.gz.asc
            name: openssl_1.1.1.orig.tar.gz.asc
            size: "488"
            md5sum: f3296150114069ea73a72eafbfdcbb295b770e7cbf3266f9590f3d0932498b3e
          - uri: http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1-1ubuntu2.1~18.04.4.debian.tar.xz
            name: openssl_1.1.1-1ubuntu2.1~18.04.4.debian.tar.xz
            size: "95260"
            md5sum: a373c2612817f3ae929d01ddb9175a6f9ab0ac28a08afe93c88df27fadcc7500

cirocosta commented 5 years ago

kind: files/v1
data:
  - name: b
    path: /dest/b
    digest: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    from_tarball:
        path: /src/archive.tgz
        digest: sha256:da10eb65191c2cbc267632b5d24ea992e322959290041560380537e156006e68
        unarchived_location: /dest/estailero-tar584446364
  - name: c
    path: /dest/c
    digest: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    from_tarball:
        path: /src/archive.tgz
        digest: sha256:da10eb65191c2cbc267632b5d24ea992e322959290041560380537e156006e68
        unarchived_location: /dest/estailero-tar584446364

kind: filesources/v1
data:
    /usr/dummy-file:
        type: git
        ref: ""
        repository: https://github.com/cirocosta/estaleiro
    /usr/test-file:
        type: git
        ref: ""
        repository: https://github.com/cirocosta/estaleiro

cirocosta commented 5 years ago

See https://github.com/cirocosta/estaleiro/blob/6b7526669d15378615b181f360a3f3d520fb5286/bom.yml

cirocosta / estaleiro

BOM #9

generation of bill of materials (bom)

the problem space

format

base image

packages

files

from steps

from tarballs