Open TerryHowe opened 1 year ago
Hi @TerryHowe ,
Thanks for the feature request. It looks like the OCI image layout #378 can cover the core ability to upload the data saved on disk to a registry as you described above.
ORAS supported specifying an authentication file when pulling/pushing files from/to the registry. Can you elaborate more on "Need to be able to specify credentials and TLS for source or destination registry"? Does it mean the ability to use certificates at path (.crt, .cert, .key) to connect to the destination registry and require HTTPS and verify certificates when interacting with the registry respectively?
Credentials I would expect similar to the docker config.json format.
I don't have any firm TLS cert requirements because there are a lot of options there, but need to support self signed certs and private CAs.
Credentials I would expect similar to the docker config.json format.
oras reuses the same credential schema of docker config.json file
need to support self signed certs and private CAs.
--ca-file
do support private CAs, and you can add self-signed cert to the CA file so self-singed certs are also supported.
With oras v1.0.0-rc.1
, this feature request is fulfilled by using oras copy --to-oci-layout
and oras copy --from-oci-layout
where OCI layout supports multiple resources in a single folder.
With
oras v1.0.0-rc.1
, this feature request is fulfilled by usingoras copy --to-oci-layout
andoras copy --from-oci-layout
where OCI layout supports multiple resources in a single folder.
oras copy
only copies one resource and it does not preserve repository name. If I want to create a mirror, particularly in air gapped environment, I want to download a long list of images and then sneaker net them to my air gapped environment and upload them to my private registry.
oras copy
... does not preserve repository name.
The repository name can be preserved if you use the registry full FQDN as the layout path, e.g. below command will download the image to an OCI layout in ${pwd}/docker.io/library/hello-world
.
oras cp docker.io/library/hello-world:latest --to-oci-layout docker.io/library/hello-world:latest --platform linux/amd64
The repository is in the relative path from docker.io
to the oci root directory hello-world
The idea of batched uploading/downloading sounds interesting to me. Providing batched operations is a brand-new feature to oras CLI so it's better to start with a design discussion.
@TerryHowe Downloading multiple resources to local filesystem is as simple as saving the following bash script as oras_mirror_backup.sh
#!/bin/bash
for ref in $@; do
echo ">>> Backing up $ref ..."
oras cp $ref --to-oci-layout $ref
done
and then do
$ ./oras_mirror_backup.sh docker.io/library/hello-world:linux docker.io/library/busybox:latest
>>> Backing up docker.io/library/hello-world:linux ...
Copied [registry] docker.io/library/hello-world:linux => [oci-layout] docker.io/library/hello-world:linux
Digest: sha256:1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
>>> Backing up docker.io/library/busybox:latest ...
Copied [registry] docker.io/library/busybox:latest => [oci-layout] docker.io/library/busybox:latest
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
As you may noticed, the blobs are not de-duplicated across the repositories. Here's an advanced script
#!/bin/bash
function extract_path {
local ref="$1"
if [[ "$ref" == *@* ]]; then
echo "${ref%@*}"
else
echo "${ref%:*}"
fi
}
# create a shared blobs folder and a repositories folder
mkdir -p blobs repositories
cd repositories
# start mirroring
for ref in $@; do
echo ">>> Backing up $ref ..."
path=$(extract_path $ref)
if [[ ! -L $path/blobs ]]; then
mkdir -p $path
ln -rs ../blobs $path/blobs
fi
oras cp $ref --to-oci-layout $ref
done
If you run the above script, you will get
$ ./oras_mirror_backup.sh docker.io/library/hello-world:linux docker.io/library/busybox:latest
>>> Backing up docker.io/library/hello-world:linux ...
Copied [registry] docker.io/library/hello-world:linux => [oci-layout] docker.io/library/hello-world:linux
Digest: sha256:1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
>>> Backing up docker.io/library/busybox:latest ...
Copied [registry] docker.io/library/busybox:latest => [oci-layout] docker.io/library/busybox:latest
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
Here's the final folder structure:
$ tree
.
├── blobs
│ └── sha256
│ ├── 04341b189be695acecd201a36cdf9dd99b8b0c338075000f555d5adf8e9c0547
│ ├── 069e43a261e5dd787655dbeba5eed96e40f4c9f80f024ecd5d2bd17aab357204
│ ├── 0af8c5262529b2acebe9e308296ea619f25b2b3b47c632f7ff154e931d18064c
│ ├── 0b41f34c76745eef3d807afe679bea40ad0d84c1f109578ff017c870f2137589
│ ├── 1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
│ ├── 1c8bbeaff20b74c3918ae3da99db0f0d8563adb33fcb346592e2882d82c28ab5
│ ├── 1d57ab16f681953c15d7485bf3ee79a49c2838e5f9394c43e20e9accbb1a2b20
│ ├── 205dae5015e78dd8c4d302e3db4eb31576fac715b46d099fe09680ba28093a7a
│ ├── 2bc9dea49d1a226db134bce761bfa89dd456109555c3ee4c490db84ad48d53b0
│ ├── 2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54
│ ├── 33450689bfb495ed64ead935c9933f1d6b3e42fe369b8de9680cf4ff9d89ce5c
│ ├── 36d89aa75357c8f99e359f8cabc0aae667d47d8f25ed51cbe66e148e3a77e19c
│ ├── 3caa6dc69d0b73f21d29bfa75356395f2695a7abad34f010656740e90ddce399
│ ├── 40d0cfd0861719208ff9f7747ab3f97844eeca509df705db44a736df863b76af
│ ├── 432f982638b3aefab73cc58ab28f5c16e96fdb504e8c134fc58dff4bae8bf338
│ ├── 46331d942d6350436f64e614d75725f6de3bb5c63e266e236e04389820a234c4
│ ├── 46758452d3eef8cacb188405495d52d265f0c3a7580dfec51cb627c04c7bafc4
│ ├── 4c45e4bb3be9dbdfb27c09ac23c050b9e6eb4c16868287c8c31d34814008df80
│ ├── 4ff685e2bcafdab0d2a9b15cbfd9d28f5dfe69af97e3bb1987ed483b0abf5a99
│ ├── 5004e9d559e7a75f42249ddeca4d5764fa4db05592a7a9a641e4ac37cc619ba1
│ ├── 5e42fbc46b177f10319e8937dd39702e7891ce6d8a42d60c1b4f433f94200bd2
│ ├── 61fff98d5ca765a4351964c8f4b5fb1a0d2c48458026f5452a389eb52d146fe8
│ ├── 66ba00ad3de8677a3fa4bc4ea0fc46ebca0f14db46ca365e7f60833068dd0148
│ ├── 688cd001103a44dc582d4fdc4647517422c0be7942c1278b5bb748395265375d
│ ├── 7050e35b49f5e348c4809f5eff915842962cb813f32062d3bbdd35c750dd7d01
│ ├── 7066d68bd2f224dbb7c3332da105b1dac81a75b47a869602096c27b6a75a525c
│ ├── 77ed5ebc3d9d48581e8afcb75b4974978321bd74f018613483570fcd61a15de8
│ ├── 7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
│ ├── 7b8b7289d0536a08eabdf71c20246e23f7116641db7e1d278592236ea4dcb30c
│ ├── 7ef0bcd6b4899cdcc5d2c4e97aba3e60a0153a3201c1a9c810ed915975f3833e
│ ├── 7f0d4fad461d1ac69488092b5914b5ec642133c0fb884539045de33fbcd2eadb
│ ├── 8f23e10f4610afdde9b856b9367742f1f5ded5c35e2aaa0630d3c5d9ebc2e4cf
│ ├── 907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754
│ ├── 93f830f96e6d4290268f3d7adb078a66ddc24c23dddbd4899fd72a8041a5a1c8
│ ├── 95a526907ab34a09463a07cd768039ac815d433029f181a7731ef8dba3095bc4
│ ├── 98c9722322be649df94780d3fbe594fce7996234b259f27eac9428b84050c849
│ ├── 995efde2e81b21d1ea7066aa77a59298a62a9e9fbb4b77f36c189774ec9b1089
│ ├── 99ee43e96ff50e90c5753954d7ce2dfdbd7eb9711c1cd96de56d429cb628e343
│ ├── 9af22d424aada215bab8c43d48ba6c8e4ddae9018628ab2098f16520bfdcd6d8
│ ├── a22ab831b2b2565a624635af04e5f76b4554d9c84727bf7e6bc83306b3b339a9
│ ├── abaa813f94fdeebd3b8e6aeea861ab474a5c4724d16f1158755ff1e3a4fde8b0
│ ├── abc70fcc95b2f52b325d69cc5c259dd9babb40a9df152e88b286fada1d3248bd
│ ├── b203a35cab50f0416dfdb1b2260f83761cb82197544b9b7a2111eaa9c755dbe7
│ ├── b3593dab05491cdf5ee88c29bee36603c0df0bc34798eed5067f6e1335a9d391
│ ├── b49eda688ce8c1226b6d7e02969f22361a8874cfee14c603e98ad855f1267a94
│ ├── b836bb24a270b9cc935962d8228517fde0f16990e88893d935efcb1b14c0017a
│ ├── b921b04d0447ddcd82a9220d887cd146f6ef39e20a938ee5e19a90fc3323e030
│ ├── bbc6052697e5fdcd1b311e0b3f65189ffbe354cf8ae97e7a55d588e855097174
│ ├── c0218de6585df06a66d67b25237bdda42137c727c367373a32639710c7a9fa94
│ ├── c3505dfdb7a6ef524d17d0ee391749f94de950c43642e3286e06172577e184a3
│ ├── c7b6944911848ce39b44ed660d95fb54d69bbd531de724c7ce6fc9f743c0b861
│ ├── db6ea0cbfcdfe2e7fff3f36b40c2c6ac27933977d71317b30c1905675ec29349
│ ├── dde8e930c7b6a490f728e66292bc9bce42efc9bbb5278bae40e4f30f6e00fe8c
│ ├── df5477cea5582b0ae6a31de2d1c9bbacb506091f42a3b0fe77a209006f409fd8
│ ├── e954aa43bc3d58a30a967d36b0b0ebf408eea4b1283106d2ca553b0243858d6b
│ ├── eb11b1a194ff8e236a01eff392c4e1296a53b0fb4780d8b0382f7996a15d5392
│ ├── f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4
│ ├── f78e6840ded1aafb6c9f265f52c2fc7c0a990813ccf96702df84a7dcdbe48bea
│ └── feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
├── oras_mirror_backup.sh
└── repositories
└── docker.io
└── library
├── busybox
│ ├── blobs -> ../../../../blobs
│ ├── index.json
│ ├── ingest
│ └── oci-layout
└── hello-world
├── blobs -> ../../../../blobs
├── index.json
├── ingest
└── oci-layout
11 directories, 64 files
/cc @sajayantony @qweeah
I was going to add optimization to this as a follow-up, just trying to keep it simple for now. This is supposed to be a POC.
Sure, everyone could write their own shell script, but there are a lot of use cases for this. I'd much rather throw a tar ball in a bucket and tell a customer to oras import
it and know it will work.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
The discussion is continued in #1366
What is the version of your ORAS CLI
0.16.0
What would you like to be added?
Ability to download and upload one or more resources to and from disk.
Why is this needed for ORAS?
In order to mirror registries, we need the ability to download one or more resources from a registry to disk. We also need the ability to upload the data saved on disk to a registry.
~/.docker/config.json
Are you willing to submit PRs to contribute to this feature?