WIP: Record (TOC digest → DiffID) mapping in BlobInfoCache

[mtrmac]

A single DiffID may map to multiple TOC digest values. Record that in BlobInfoCache, and use it for layer reuse.

Also prefer reusing even TOC-matched layers by DiffID, when available.

@giuseppe I’d appreciate a preliminary review of the new logic; see individual commits.

Draft: The BlobInfoCache implementations don’t actually store/record any data yet — so this is obviously completely untested.

[mtrmac]

To test:

Before:

# podman rmi alpine level1 level9
# rm -f /var/lib/containers/cache/blob-info-cache-v1.sqlite 
# podman pull quay.io/libpod/alpine
# podman --log-level=debug push --compression-format zstd:chunked --compression-level 1 --force-compression quay.io/libpod/alpine localhost:50000/level1
## Even better would be to use two different destination registries, to be 100% certain the blobs are not reused
## (right now they are not reused, but we’ll fix that):
# podman--log-level=debug push --compression-format zstd:chunked --compression-level 9 --force-compression quay.io/libpod/alpine localhost:50000/level9
## Note the compressed digest, and TOC digest, values:
# skopeo inspect --raw docker://localhost:50000/level1 | jq .
# skopeo inspect --raw docker://localhost:50000/level9 | jq .
## No DigestTOCUncompressedPairs entries:
# sqlite3 /var/lib/containers/cache/blob-info-cache-v1.sqlite .dump 
# podman rmi alpine level1 level9
## Triggers a partial pull: "Applying differ in …":
# podman --log-level=debug pull localhost:50000/level1
## Triggers a partial pull: "Applying differ in …"
# podman --log-level=debug pull localhost:50000/level9 
## level1 and level9 have different image IDs:
# podman images 
## Contains two copies of the layer, with the same expected-layer-diffid
# jq . < /var/lib/containers/storage/overlay-layers/layers.json ```

After:

• DigestTOCUncompressedPairs contains 2 records
• Pull of level1 triggers a partial pull (creating a layer with known TOC digest and uncompressed digest)
• Pull of level9 reuses the layer (by BIC compressed -> uncompressed mapping)
• FIXME: the layer is shared, but the image not yet - the hasLayerPulledByTOC code path is wrong