Complete Ingestion of Docker Images

application-research / ipcr

The toolkit to pack, ship, store, and deliver container content on Estuary/FIlecoin/IPFS

Apache License 2.0

8 stars 1 forks source link

Open 10d9e opened 2 years ago

10d9e commented 2 years ago

Create jobs to continually ingest images from target docker registries. Given the volume of images, the ingestor should have the following:

automatic error recovery/retry logic
persistent repository upload tracker - badgerdb or similar db to track the progress of uploads
explore "batched" upload outside of registry (may not be possible, but worth looking at)

10d9e commented 2 years ago

This process is quite slow through the native docker client. There may be a better way to rewrite large-scale ingestion