Harbor, Ground Control and Satellite interaction

Vad1mo commented 1 month ago

Based on the discussion last week, I want to summarize the responsibilities of different software components, and what effect it has on the users' interaction with those systems.

User Story

As I user of Satellite, I intend to continue using harbor as the primary image management solution. I would like to use the existing replication capabilities of Harbor to select what images should be present on each edge device or edge group.

Software Components

Harbor: Responsible for managing artifacts.
Ground Control: responsible for managing edge devices, registration, grouping, heartbeat, etc.
Satellite: The local registry. Connected to Ground Control for registration, grouping, heartbeat. Harbor for fetching artifacts and state.

Implementation Harbor Side

The user manages the artifacts in Harbor and decides which artifacts should be present on the edge. The goal is to reuse all the existing Harbor capabilities, and Harbor is extended with a new Registry Provider. The user will be using existing replication rules to select which artifacts should be present in which group. Instead of pushing artifacts down to the edge, the new registry provider will make a list and send this list to ground control.

Given that list, the edge device (satellite) will be pulling the images to its side.

Essentially, the registry provider that instead of actively moving images creates a list of those images and pushes that list to the ground control which then pushed it edge, which then downloads the images

Replication and Grouping

Currently, replication is a 1:1 relationship with one source and one destination registry. However, in the satellite use case we are not replication to each edge device (via ground control) but to each group of devices.

So there is the requirement to be able to address multiple edge devices. This can be done in two ways, at the endpoint or namespace level.

1) Every Group in Ground Control is a Registry Endpoint in Harbor 2) Every Group in Ground Control is a destination namespace inside the Replication Rule

So far, the decision was made to execute option no. 2

Further Steps

TODO

[ ] Create new Harbor registry provider
[ ] Registry provider is creating the state list for that group
[ ] Registry provider send the state file to ground control (REST endpoint on ground control to be defined.
[ ] Registry Endpoint URL, Access ID, Secret describes the REST API of ground control.

Sample Edge State File

{
  "artifacts": [
    {
      "deleted": false,
      "digest": "sha256:6a2f8b28e45c4adea04ec207a251fd4a2df03ddc930f782af51e315ebc76e9a9",
      "repository": "blabla/nginx",
      "tag": [
        "mainline-alpine",
        "alpine3.18"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:14c426f3dc89970649e2376577484e3d3f55930b6838dd78addab166c50a557d",
      "repository": "blabla/nginx",
      "tag": [
        "1-alpine-perl",
        "1.25-alpine3.18-perl",
        "alpine-perl",
        "alpine3.18-perl",
        "1-alpine3.18-perl",
        "mainline-alpine-perl",
        "1.25-alpine-perl",
        "1.25.4-alpine3.18-perl",
        "1.25.4-alpine-perl"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:d4ade3639c27579321046d78cc44ec978cea8357d56932611984f601d27e30ac",
      "repository": "blabla/alpine",
      "tag": [
        "latest"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:6a2f8b28e45c4adea04ec207a251fd4a2df03ddc930f782af51e315ebc76e9a9",
      "repository": "library/nginx",
      "tag": [
        "alpine3.18",
        "mainline-alpine"
      ],
      "type": "IMAGE"
    },

    {
      "deleted": false,
      "digest": "sha256:6a2f8b28e45c4adea04ec207a251fd4a2df03ddc930f782af51e315ebc76e9a9",
      "repository": "public/nginx",
      "tag": [
        "mainline-alpine",
        "alpine3.18"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:14c426f3dc89970649e2376577484e3d3f55930b6838dd78addab166c50a557d",
      "repository": "public/nginx",
      "tag": [
        "1-alpine-perl",
        "1.25-alpine3.18-perl",
        "alpine-perl",
        "mainline-alpine3.18-perl",
        "alpine3.18-perl",
        "1-alpine3.18-perl",
        "mainline-alpine-perl",
        "1.25-alpine-perl",
        "1.25.4-alpine3.18-perl",
        "1.25.4-alpine-perl"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:d4ade3639c27579321046d78cc44ec978cea8357d56932611984f601d27e30ac",
      "repository": "public/alpine",
      "tag": [
        "latest"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:6a2f8b28e45c4adea04ec207a251fd4a2df03ddc930f782af51e315ebc76e9a9",
      "repository": "test/nginx",
      "tag": [
        "mainline-alpine",
        "alpine3.18"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:14c426f3dc89970649e2376577484e3d3f55930b6838dd78addab166c50a557d",
      "repository": "test/nginx",
      "tag": [
        "1-alpine-perl",
        "1.25-alpine3.18-perl",
        "alpine-perl",
        "mainline-alpine3.18-perl",
        "alpine3.18-perl",
        "1-alpine3.18-perl",
        "mainline-alpine-perl",
        "1.25-alpine-perl",
        "1.25.4-alpine3.18-perl",
        "1.25.4-alpine-perl"
      ],
      "type": "IMAGE"
    },
    {
      "deleted": false,
      "digest": "sha256:d4ade3639c27579321046d78cc44ec978cea8357d56932611984f601d27e30ac",
      "repository": "test/alpine",
      "tag": [
        "latest"
      ],
      "type": "IMAGE"
    }
  ],
  "registry": "test"
}

Next Steps and Use Cases

Once ground control has the state for the edge group, it will create an OCI state artifact and push it to Harbor. The satellite edge groups are periodically polling Harbor for new OCI state artifacts, and once they find any change will pull that artifact, verifies and extract it and download the artifacts listed in the edge state file.

Alternative Additional Considerations

Our goal is to transfer and package the state as an OCI artifact. Given that ground control should create such an OCI state artifact and push it back to the registry, this process can also be created by the registry provider itself, which is located on the harbor side.

bupd commented 1 month ago

I really appreciate the overall direction of this idea.

I have a few thoughts and questions regarding the structure and assignment:

Current Data Structure Breakdown:

Groups: Contains sub-groups, satellites, and artifacts.
Artifacts: Can be tagged with labels for easy identification.
Satellite: Contains groups, labels, and associated artifacts.
Relationship:
Groups-->Artifacts (one to many)
Artifacts--> Labels(one to many)
Satellite --> Groups, Labels, Artifacts (many to many)

In this structure, a satellite will contain:

Labels: To have artifact.
Groups: Which can have their own associated artifacts.
Artifacts: Directly assigned to the satellite.

Questions & Clarifications:

Are we only supporting image assignment at the group level?
- If yes, this will simplify the image assigning process to just groups and labels.
What if we need to assign images to individual satellites?
- While I understand the flexibility this provides, I suggest we consider removing the option to assign artifacts directly to satellites. Since this can get complicated for users managing satellites on a large scale.
- Let's clarify this, Do we want to limit image assignment to groups only, or are we allowing satellite-specific image assignments as well?

Overall, I assume that aside from the image assignment changes, the rest of the processes such as satellite registration, state artifact creation etc., will remain unchanged.

Vad1mo commented 1 month ago

fixed by https://github.com/container-registry/harbor-satellite/pull/55

Vad1mo commented 3 weeks ago

@Mehul-Kumar-27, can you please paste here an excerpt from the state file, so that @tpoxa can take a look on what is wrong there!

Mehul-Kumar-27 commented 3 weeks ago

This is how the state json looks currently

{
  "registry": "Satellite",
  "artifacts": [
    {
      "repository": "satellite-test-group-state/alpine",
      "tag": [
        "latest"
      ],
      "labels": null,
      "type": "IMAGE",
      "digest": "sha256:9cee2b382fe2412cd77d5d437d15a93da8de373813621f2e4d406e3df0cf0e7c",
      "deleted": false
    },
    {
      "repository": "satellite-test-group-state/alpine",
      "tag": [
        "edge"
      ],
      "labels": null,
      "type": "IMAGE",
      "digest": "sha256:4b27406f6c4ea07b596d7a0ee25e4f9c7b2152ab7ffbbb1817e036d727eabba1",
      "deleted": false
    },
    {
      "repository": "satellite-test-group-state/postgres",
      "tag": [
        "latest"
      ],
      "labels": null,
      "type": "IMAGE",
      "digest": "sha256:dde924f70bc972261013327c480adf402ea71487b5750e40569a0b74fa90c74a",
      "deleted": false
    }
  ]
}

The issue lies in the repository field.

For example i have the image under a project which has the structure like this library/ ├── busybox/ └── hello-world/

For satellite to fetch the busybox image it would require the repository path as library/busybox but in the current implementation of the state json it's path would appear as satellite-test-group-state/busybox where satellite-test-group-state is the name of the group which we add while creating the replication rule as namespace.

Cc. @Vad1mo , @tpoxa , @bupd

container-registry / harbor-satellite