search

admiraltyio / admiralty

A system of Kubernetes controllers that intelligently schedules workloads across clusters.
https://admiralty.io
Apache License 2.0
674 stars 87 forks source link

Need advice how to connect two OpenShift clusters using Admiralty #129

Closed hfwen0502 closed 10 months ago

hfwen0502 commented 2 years ago

My initial experiment was to have two OpenShift clusters, one is acting as the management cluster and the other is the cluster for running workloads. I think I got some success as I can see the virtual node from the mgmt. cluster:

[root@hf-ocp-login1 ~]# oc get nodes NAME STATUS ROLES AGE VERSION 10.242.65.11 Ready master,worker 46h v1.20.0+bbbc079 10.242.65.12 Ready master,worker 46h v1.20.0+bbbc079 10.242.65.13 Ready master,worker 46h v1.20.0+bbbc079 admiralty-default-ocp1 Ready cluster 62m

However, the job is staying in the pending state, not being able to be scheduled by Admiralty. I have seen a couple of issues based on the logs from Admiralty pods and I think the issues are related to cross-cluster authentication. In the step 4 below, clusters[0].cluster.server is for the API server for the workload cluster right? In addition to $IP, I would need to change the port number 6443 to the api server in my workload cluster. Is that correct?

  # iii.
  IP=$(docker inspect $CLUSTER_NAME-control-plane \
    --format "{{ .NetworkSettings.Networks.kind.IPAddress }}")

  # iv.
  CONFIG=$(kubectl --context kind-$CLUSTER_NAME config view \
    --minify --raw --output json | \
    jq '.users[0].user={token:"'$TOKEN'"} | .clusters[0].cluster.server="https://'$IP':6443"')

  # v.
  kubectl --context kind-cd create secret generic $CLUSTER_NAME \
    --from-literal=config="$CONFIG"

I am using the api server address provided by oc config view using the port number 30571 instead of 6443.

[root@hf-ocp-login1 ~]# oc config view
apiVersion: v1
clusters:
- cluster:
    server: https://c100-e.eu-gb.containers.cloud.ibm.com:30571
  name: hf-ocp1-eu-zone2/c6dumb0l00qnml047mag
hfwen0502 commented 2 years ago

The errors I got from the Admiralty agent pod (image: quay.io/admiralty/multicluster-scheduler-agent:0.14.1) are shown as below:

E1124 22:14:03.945330       1 reflector.go:178] k8s.io/client-go@v0.18.9/tools/cache/reflector.go:125: Failed to list *v1alpha1.ClusterSummary: Get "https://c100-e.eu-gb.containers.cloud.ibm.com:30571/apis/multicluster.admiralty.io/v1alpha1/clustersummaries?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
tonycoder-hub commented 2 years ago

I think maybe I am facing the same question as you do. If I may ask, what's your solution?