search

anchore / grype-db

Apache License 2.0
31 stars 13 forks source link

All DB updates from Daily DB publisher uses the same built date since job #467? #315

Closed mblixter closed 1 month ago

mblixter commented 1 month ago

Since Daily DB builder job #467 all subsequent jobs promote the DB with the same built date 2024-05-17T01:31:32Z making grype scans fail due to the DB being to old?

From Job #471 Entry(built='2024-05-17T01:31:32Z', version=3, url='https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v3_2024-05-17T01:31:32Z_1716180320.tar.gz', checksum='sha256:a559f9e84f8352a0419281b6e7e2d965d65b35112be172e1b24c9f5842950006')

From Job #470 Entry(built='2024-05-17T01:31:32Z', version=4, url='https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v4_2024-05-17T01:31:32Z_1716094053.tar.gz', checksum='sha256:be8159d82fec4eb0f18b4538a22b763a7766ecb448d6cef75430a8d51c31c262')

wagoodman commented 1 month ago

There is another DB build running now that should have updated timestamps. Here's what was going on:

The built field is poorly named -- it is really the age of the data that the DB is populated from. We gather data from several upstream sources, so we use the oldest timestamp amongst all of the timestamps from each provider. One of the upstream providers this weekend had a data outage over the weekend and a holiday, so the DB built field was held back to when this last provider was working. Luckily they restored their service this morning and a new DB should be going out shortly with the latest data.

We've already started making changes to start including more metadata about providers (such as data freshness for each) and will be including this information in the DB itself (probably eliminating the need to have a single old timestamp representing the DB). This will most likely land in the v6 DB schema changes (which design work has only just begun).

wagoodman commented 1 month ago

Confirmed we're now seeing the correct timestamps:

grype db list -o json | jq | head -n 20
[
  {
    "built": "2024-05-21T10:13:53Z",
    "version": 5,
    "url": "https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2024-05-21T10:13:53Z_1716302442.tar.gz",
    "checksum": "sha256:463f62c79b5c95f914d5938f3a116ff329b854a715243ac29f64d08c5a8c048d"
  },
  {
    "built": "2024-05-17T01:31:32Z",
    "version": 5,
    "url": "https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2024-05-17T01:31:32Z_1716007601.tar.gz",
    "checksum": "sha256:96a5396fc536f1d239960a3719114d198a7e191485d5e87a5232c22dfc99cafb"
  },
  {
    "built": "2024-05-17T01:31:32Z",
    "version": 5,
    "url": "https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2024-05-17T01:31:32Z_1716094011.tar.gz",
    "checksum": "sha256:ef89a271e05c74ba54c2d756386dc2b0386827f370612dd632bb559137ef8f93"
  },
...