Replication between two k8s clusters.

[one1zero1one]

Hello,

I deployed Kraken using the helm chart on two different kubernetes clusters, and am trying to set up image replication between clusters.

Clusters are set up with kind on osx.

I am using two different chart value files to deploy on each cluster: kind-1 and kind-2 I have tweaked a bit the chart to allow for validating kraken functionality in each cluster (mostly added ingress and customisable port to the proxy).

With that, I am able to successfully pull from the agent and push/pull from the proxy, from the osx (after setting 127.0.0.1 kind-1-proxy.kind kind-2-proxy.kind in /etc/hosts). The blob and tags are on present on the testfs.

	kind-1	kind-2
agent	docker pull localhost:30081/library/<image>	docker pull localhost:30082/library/<image>
proxy	docker push kind-1-proxy.kind/<image>	docker push kind-2-proxy.kind:81/<image>

My goal with this test is to setup replication such that when I push to the proxy of kind-1 the image to be replicated to kind-2. So far I have added the remotes to kind-1 via the chart, which ends up in the config like this

remotes:
  "kind-2-build-index.kind:81":
  - .*

I have also made sure pods in kind-1 cluster can reach the kind-2 ingress (added 172.17.0.1 kind-1-build-index.kind kind-2-build-index.kind to /etc/hosts).

When I push an image (ctop:latest) to the proxy , I can see this in the logs of kraken in kind-1

kraken-origin-6f694df888-kdfb7 main 2020-07-07T21:37:05.873Z    INFO    handler/handler.go:98   409 POST /namespace/ctop:latest/blobs/sha256:6532064fcb4a37b37bc507d60b064399d55d74d1277a6183732abfbbd9f9e633/uploads
kraken-origin-6f694df888-kdfb7 main 2020-07-07T21:37:05.884Z    INFO    handler/handler.go:98   409 POST /namespace/ctop:latest/blobs/sha256:ab3568c603397a277a3411733685a82afe2388cb6c72832273ddfd417ec77040/uploads
kraken-build-index-7785d4d45-hn9hr main 2020-07-07T21:37:05.894Z    ERROR   persistedretry/manager.go:207   Task failed: put and replicate tag: PUT http://kind-2-build-index.kind:81/tags/ctop:latest/digest/sha256:ab3568c603397a277a3411733685a82afe2388cb6c72832273ddfd417ec77040?replicate=true 500: resolve dependencies: download blob: blob not found   {"task": "tagreplication.Task(tag=ctop:latest, dest=kind-2-build-index.kind:81)", "failures": 271}

while in kind-2 build index can't find the blob.

kraken-build-index-5f4ff66666-lcnbn main 2020-07-07T21:37:21.889Z   INFO    handler/handler.go:98   500 PUT /tags/ctop:latest/digest/sha256:ab3568c603397a277a3411733685a82afe2388cb6c72832273ddfd417ec77040 resolve dependencies: download blob: blob not found

As far as I can understand from Kraken architecture in step 5 the index should get back the origin address, to be able to then replicate the blob.

My assumption (based on limited understanding) is that origins and index from both k8s clusters need to form a cluster of their own (across the k8s clusters) for this to work. The _helpers.tpl seems to suggest that for origins and build-index there is a way to specify a cluster member list.

Thank you for any suggestions on if the above setup (or any similar) is possible - and how would the config look.