[feature] Template option for displaying manifest before sync

[clemenko]

When using hauler store sync --products... it can potentially download in excess of 40gb of data. Having a template function to show the manifest BEFORE actually downloading would allow teams to choose, check in to version control, and compare image lists they may already have.

-

Describe Proposed Solution(s):

create a flag --create-manifest for exporting the manifest yaml. Either to stdout or to a file. hauler store sync --products rancher=2.8.6 --create-manifest This will give teams the ability to review what images are being pulled. Check the manifest into version control. And potentially trim down the manifest to remove images that are not needed. aka removing longhorn, neuvector, isitio and the likes.

as an example

  curl -sL https://github.com/rancher/rancher/releases/download/$RANCHER_VERSION/rancher-images.txt -o hauler_temp/orig-rancher-images.txt
  sed -E '/neuvector|minio|gke|aks|eks|sriov|harvester|mirrored|longhorn|thanos|tekton|istio|hyper|jenkins|windows/d' hauler_temp/orig-rancher-images.txt > hauler_temp/cleaned-rancher-images.txt

  # capi fixes
  grep cluster-api hauler_temp/orig-rancher-images.txt >> hauler_temp/cleaned-rancher-images.txt
  grep kubectl hauler_temp/orig-rancher-images.txt >> hauler_temp/cleaned-rancher-images.txt

  # get latest version
  for i in $(cat hauler_temp/cleaned-rancher-images.txt|awk -F: '{print $1}'); do 
    grep -w "$i" hauler_temp/cleaned-rancher-images.txt | sort -Vr| head -1 >> hauler_temp/rancher-unsorted.txt
  done

-

Describe Possible Alternatives:

current alternative is

hauler store add image rgcrprod.azurecr.us/hauler/rancher-manifest.yaml:v2.8.5
hauler store extract hauler/rancher-manifest.yaml:v2.8.5

which actually starts to download before creating the local manifest.

-

Additional Context:

-

[wcrum]

Doing some research into this, have a working demo of calculating total image size and total file size (helm + files). For the images you can calculate the total download size by iterating through the OCI Manifest and taking the sum of the layer size, this is a compressed total gzip, this would not account for total size taken up within the store, which potentially could lead to a disk usage issue.

For content pulled from the web you can check content-length and just sum that up.

Would this suffice @clemenko?

> hauler store sync -f manifest.yaml --check
+------------------------+-------+----------+----------+---------+
|            Total Download Size for 'manifest.yaml'             |
+------------------------+-------+----------+----------+---------+
| REFERENCE              | TYPE  | PLATFORM | # COUNT  | SIZE    |
+------------------------+-------+----------+----------+---------+
| images                 | image | -        |       12 | 842 kB  |
| files                  | file  | -        |        5 | 6.2 MB  |
| helm charts            | helm  | -        |        3 | 5.4 MB  |
+------------------------+-------+----------+----------+---------+
|                                             TOTAL    | 12.3 MB |
+------------------------+-------+----------+----------+---------+

[clemenko]

potentially. the goal is to be able to create a reusable manifest from --products to then remove or add items.

[zackbradys]

I think this would be a three part transition that would cover three open feature requests, including this one. Let me do a bit of thinking with the rest of the team and get back to you both next week.

I like the initial stab at it @wcrum!