Proposal: Integrate trusted cloud-native registry Harbor with Dragonfly to provide a joint image management and distribution solution to support containerized environments

[steven-zou]

STATUS: [INPROGRESS]

Integrate trusted cloud-native registry Harbor with Dragonfly to provide a joint image management and distribution solution to support containerized environments.

Backgrounds:

Harbor: Project Harbor is an open source trusted cloud-native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity, and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries and also offers advanced security features such as user management, access control, and activity auditing. For more details, please refer to README.

Dragonfly: Dragonfly is an intelligent P2P based file distribution system. It aims to resolve issues related to low-efficiency, low-success rate and a waste of network bandwidth in file transferring process. Especially in large-scale file distribution scenarios such as application distribution, cache distribution, log distribution, image distribution, etc. For more details, please refer to README

Motivations:

With the emergence and development of Kubernetes, it's becoming possible to run and operate large-scale containerized applications and services in enterprise environments. Meanwhile, there are still existing big challenges which cannot be ignored. How to securely and effectively manage the lots of container images produced in the enterprise organizations and distribute them to the large-scale runtimes with less time and efforts when starting applications or services on demand. To address the above challenge, we should build a joint solution from the open source trust cloud-native registry Harbor and the open source intelligent P2P based file distribution system Dragonfly.

These two open sourced projects have very obviously complementary advantages to each other and the joint solution will definitely expand the scenarios of image lifecycle management and improve the securities, reliabilities, and efficiencies.

Idea:

• The integration should be a loose couple way, by calling related APIs to complete the required work. The system admin of Harbor registry can configure the related options to enable the API calling from Dragonfly side. The options may include but not limit the following ones:

  • API endpoint
  • API access token or required credentials
  • Possible calling policies or automation rules etc.

• The integrated configurations can be verified to make sure the connection between the two systems is not broken by testing or ''dry run" etc.

• The images are produced by CI/CD pipeline or any other ways and pushed to the Harbor registry. The newly pushed images can be marked with labels automatically or manually. In addition, the admin of registry can also scan the images to make sure it's secure. Of course, the admins can do any other management work if they want.

• The admin of registry can select any ready image to promote it to the supervise node of Dragonfly P2P network for the upcoming image pulling requests to improve the distribution performance. The promote action can be triggered by clicking button or auto-triggered by pre-configured rules/policies (If match some conditions, then promote it).

• Then if the containerized environments need to pull that image, the Dragonfly will help to distribute it to the nodes by layers via the P2P network.

Basic Workflow:

Architecture:

An architecture design based on the above draft idea:

The components with light blue background are the new things need to be implemented.

• The controller provides related API methods to handle the overall workflow
• The config will handle loading and saving of the related configurations based on the existing Harbor configuration service
• The policy engine handles the CRUD of policy as well as the evaluation of the policies
• Hook controller is designed to take charge of event hook related things
• Image distribution driver will define as an interface to provide the related methods to talk to the Dragonfly API
• API from Dragonfly side provides the required capabilities of publishing images to the supervise node and returning necessary status info and/or metrics if required

Followups:

• Discuss the idea and the draft design
• Confirm the Dragonfly API capabilities
• Make the development plan
• Setup the virtual team

[lowzj]

Very looking forward to the integration of Dragonfly and Harbor.

In order to complete the task of publishing images to the SuperNode, Dragonfly's internal workflow is:

1. gets the publishing image task from Image Distribution Driver
2. gets the URLs of all layers of the image from container image registry, Harbor Registry
3. sends tasks to all SuperNodes of Dragonfly to pre-download the image's layers, and periodically records the task status for query
4. then if anyone want to pull that image, Dragonfly will help him to download the image by layer via P2P network established by SuperNode

So there are some features need to be completed in Dragonfly:

• Authentication mechanism ensures that the caller is trusted
• API set provides the capabilities to publish images to the SuperNode of Dragonfly
• DFDaemon(the proxy deployed on every machines) supports pulling private container images

And something need to be confirmed:

• which API provided by harbor that Dragonfly can use it to get URLs of all layers of the image
  • I think the Image Distribution Driver is the connector between container image registry and file distribution system. IMOP, it's better to use pub-sub pattern and to be an independent component. In this way, we can easily reach a very loosely coupling.

[lowzj]

I drew a new architecture design graph to add some new components of dragonfly need to be implemented.

• Dragonfly Controller provides required API for harbor to publish images. It also need to implement authentication/rate limiting mechanism to ensure security.
• Heater includes image heater and file heater that providing the ability to warm-up container images and files. It analyzes the layers of images and downloads them from harbor registry to the SuperNode local disk.

[steven-zou]

I think the following diagram is also a good reference for image distribution:

[lowzj]

Todo List of Dragonfly

• [x] 08.15 the specification of API between Dragonfly and Harbor is completed
  • check verify the connection between Dragonfly and Harbor
  • heat send a image preheating task to Dragonfly
  • query view the status of the preheating task
• [ ] 09.07 dfdaemon supports to download private container image
• [ ] 10.12 the API of dragonfly controller and heater module are completed
• [ ] 10.19 integrating demo between Dragonfly and Harbor is completed

[lowzj]

08.15 the specification of API between Dragonfly and Harbor

document: https://github.com/alibaba/Dragonfly/blob/master/docs/en/preheat.md

[steven-zou]

@lowzj

About API /api/preheat, could we make it compatible with registry API? Harbor is a registry, that means a client can use the standard docker client or registry API to get the image content.

Seems the designed API needs a new Harbor API to do that. If I'm mistaken, please correct me.

https://docs.docker.com/registry/spec/api/#pulling-an-image

[lowzj]

@steven-zou

There is no need to create a new Harbor API. Firstly, Dragonfly assembles the minifest url according to the registry API spec and param url of /api/preheat, and fetches minifest to get all the urls of image layers. Then Dragonfly downloads all the layers from Harbor.

[steven-zou]

@lowzj

So, I think the following example should be ok. One question, why is the header an array? Why not use a map?

{
  "type": "image",
  "url": "https://<harbor_hostname>/v2/library/redis/manifests/latest",
  "header": ["Authorization: Bearer <TOKEN>"]
}

[lowzj]

@steven-zou

I think the following example should be ok.

I think the url could be image url: <harbor_host>/<image_name>:<image_tag>. And the internal steps of Dragonfly could be:

• construct manifest_url:

  https://<harbor_host>/v2/<image_name>/manifests/<image_tag>

• pull the manifest of the image from manifest_url
• get thefsLayers from manifest and construct layer_url of each layer:

  https://<harbor_host>/v2/<name>/blobs/<digest>

• request these layer_urls above to handle any redirection response to get real downloading urls
• supernodes use these real downloading urls to preheat layers of this image

But I'm not sure about that whether the header is enough for authentication of these steps.

---

why is the header an array? Why not use a map?

There may be multiple message-header fields with the same filed-name in HTTP headers. If use a map, these header fields should be combined into one and each field-value should be separated by comma, like this "field-name: field-value1, field-value2,...".

Using a map may be more convenient than using an array in practice. And the multiple fields with same filed-name is not recommend. I will change the type of header to map.

[steven-zou]

@lowzj

Ok, got it. But please be aware that the <image_name> in harbor has a prefix of a project name like library/redis. You need to take care of that.

[gitzl]

Does Dragonfly integrate with Harbour support HTTPS and do the work now? If we want to modify the source code support HTTPS need to do those work and attention? Very grateful

[perriea]

An update on this issue ?

[allencloud]

An update on this issue ?

It works with Dragonfly since Dragonfly has already supported the preheat API. @perriea

While we have worked out a demo for integration of Harbor and Dragonfly. But I am not sure if the Harbor side has made a plan to release the work. @steven-zou

[datavisoryushuzhang]

Does the preheat API support private Harbor images? (which need docker login)

[allencloud]

Does the preheat API support private Harbor images? (which need docker login)

Yes, it can. You can add the login credentials in the headers. @datavisoryushuzhang

[jessiezhou0424]

when will dragonfly+harbor demo release, can't wait to use it!