ANP cluster is inaccessible, please check authorization or network /proxy/api 503 Service Unavailable

What happened:

使用ANP pull 模式

kubectl get cluster --kubeconfig karmada-apiserver.config
NAME      VERSION   MODE   READY   AGE
member1   v1.19.6   Pull   True    13m
member2   v1.19.6   Pull   True    9m29s

karmada-agent 已经加了cluster-api-endpoint,proxy-server-address

      - command:
        - /bin/karmada-agent
        - --karmada-kubeconfig=/etc/kubeconfig/kubeconfig
        - --cluster-name=member1
        - --cluster-api-endpoint=https://member-ip:6443
        - --proxy-server-address=http://proxy-ip:8088
        - --cluster-status-update-frequency=10s
        - --v=4

karmadactl get pod --kubeconfig karmada-apiserver.config
Error: [cluster(member1) is inaccessible, please check authorization or network, cluster(member2) is inaccessible, please check authorization or network]

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Karmada version: 1.2.1
kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version): karmadactl version: version.Info{GitVersion:"v1.2.1", GitCommit:"de4972b74f848f78a58f9a0f4a4e85f243ba48f8", GitTreeState:"clean", BuildDate:"2022-07-14T09:33:33Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
Others:

Could you please execute karmadactl get pod --kubeconfig karmada-apiserver.config -v=6 to get more logs.

这个地方报错了

karmadactl get pod --kubeconfig karmada-apiserver.config -v=6

.....
I1011 11:51:47.451021   29996 loader.go:372] Config loaded from file:  karmada-apiserver.config
I1011 11:51:47.451114   29996 loader.go:372] Config loaded from file:  karmada-apiserver.config
I1011 11:51:47.451926   29996 round_trippers.go:553] GET https://xxxxx:5443/apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api 404 Not Found in 0 milliseconds
I1011 11:51:47.452327   29996 round_trippers.go:553] GET https://xxxxx:5443/apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api 404 Not Found in 0 milliseconds
Error: [cluster(member1) is inaccessible, please check authorization or network, cluster(member2) is inaccessible, please check authorization or network]

根据 @631068264 在微信群里的信息来看，可能是没有安装karmada-aggregated-apiserver.

请在安装Karmada的集群上查看一下安装的组件？like:

# kubectl config use-context karmada-host
# kubectl get deployments.apps -n karmada-system 
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
karmada-aggregated-apiserver          2/2     2            2           23h
karmada-apiserver                     1/1     1            1           23h
karmada-controller-manager            2/2     2            2           23h
karmada-descheduler                   2/2     2            2           23h
karmada-kube-controller-manager       1/1     1            1           23h
karmada-scheduler                     2/2     2            2           23h
karmada-scheduler-estimator-member1   2/2     2            2           23h
karmada-scheduler-estimator-member2   2/2     2            2           23h
karmada-scheduler-estimator-member3   2/2     2            2           23h
karmada-search                        2/2     2            2           23h
karmada-webhook                       2/2     2            2           23h

有安装的 @RainbowMango

kubectl get pods -n karmada-system
NAME                                               READY   STATUS    RESTARTS   AGE
etcd-0                                             1/1     Running   0          83m
karmada-aggregated-apiserver-94f48cb85-8fqsr       1/1     Running   2          83m
karmada-apiserver-7cdb6b4f56-vsz5l                 1/1     Running   0          83m
karmada-controller-manager-99bdbb5cf-vp9qf         1/1     Running   2          83m
karmada-kube-controller-manager-54b85f668b-4nsqj   1/1     Running   2          83m
karmada-proxy-server-5777b44fb5-h4w7x              1/1     Running   0          83m
karmada-scheduler-c5b9fb5b6-jp6rg                  1/1     Running   0          83m
karmada-webhook-76b5cf9f65-f555s                   1/1     Running   2          83m

@631068264 通过什么方式安装的karmada呢？顺便可以查看ANP组件的日志，如果请求经过了ANP，会有日志的。

helm chart 安装 @lonelyCZ
karmada-0.0.5.tgz 做了一定魔改，主要是把anp 部分塞进去

host 安装参数

member 安装参数

请求经过了ANP，是什么请求？karmadactl get pod --kubeconfig karmada-apiserver.config -v=6 吗？没什么特别的日志

ANP proxy server

I1011 03:05:53.698416       1 options.go:107] ServerCert set to "/var/certs/server/server-cert.crt".
I1011 03:05:53.698556       1 options.go:108] ServerKey set to "/var/certs/server/server-key.key".
I1011 03:05:53.698563       1 options.go:109] ServerCACert set to "/var/certs/server/server-ca-cert.crt".
I1011 03:05:53.698568       1 options.go:110] ClusterCert set to "/var/certs/server/cluster-cert.crt".
I1011 03:05:53.698574       1 options.go:111] ClusterKey set to "/var/certs/server/cluster-key.key".
I1011 03:05:53.698581       1 options.go:112] ClusterCACert set to "/var/certs/server/cluster-ca-cert.crt".
I1011 03:05:53.698587       1 options.go:113] Mode set to "http-connect".
I1011 03:05:53.698593       1 options.go:114] UDSName set to "".
I1011 03:05:53.698598       1 options.go:115] DeleteUDSFile set to false.
I1011 03:05:53.698603       1 options.go:116] Server port set to 8090.
I1011 03:05:53.698611       1 options.go:117] Agent port set to 8091.
I1011 03:05:53.698616       1 options.go:118] Admin port set to 8095.
I1011 03:05:53.698622       1 options.go:119] Health port set to 8092.
I1011 03:05:53.698628       1 options.go:120] Keepalive time set to 1h0m0s.
I1011 03:05:53.698645       1 options.go:121] Frontend keepalive time set to 1h0m0s.
I1011 03:05:53.698651       1 options.go:122] EnableProfiling set to false.
I1011 03:05:53.698657       1 options.go:123] EnableContentionProfiling set to false.
I1011 03:05:53.698663       1 options.go:124] ServerID set to a93a9fe0-addc-42cf-a8f7-446682d670f8.
I1011 03:05:53.698674       1 options.go:125] ServerCount set to 1.
I1011 03:05:53.698679       1 options.go:126] AgentNamespace set to "".
I1011 03:05:53.698685       1 options.go:127] AgentServiceAccount set to "".
I1011 03:05:53.698690       1 options.go:128] AuthenticationAudience set to "".
I1011 03:05:53.698696       1 options.go:129] KubeconfigPath set to "".
I1011 03:05:53.698701       1 options.go:130] KubeconfigQPS set to 0.000000.
I1011 03:05:53.698711       1 options.go:131] KubeconfigBurst set to 0.
I1011 03:05:53.698719       1 options.go:132] ProxyStrategies set to "destHost".
I1011 03:05:53.698773       1 server.go:90] Starting frontend server for client connections.
I1011 03:05:53.699381       1 server.go:102] Starting agent server for tunnel connections.
I1011 03:05:53.699869       1 server.go:107] Starting admin server for debug connections.
I1011 03:05:53.699968       1 server.go:112] Starting health server for healthchecks.
I1011 03:09:08.945111       1 server.go:616] "Connect request from agent" agentID="0de8e8bd-2e3e-47cf-9f06-419e534a536f"
I1011 03:09:08.945228       1 backend_manager.go:191] "Register backend for agent" connection=&{ServerStream:0xc000254000} agentID="member1 IP"
I1011 03:13:15.755471       1 server.go:616] "Connect request from agent" agentID="8ee90d92-389c-464e-a02e-d565db7347f7"
I1011 03:13:15.755544       1 backend_manager.go:191] "Register backend for agent" connection=&{ServerStream:0xc00028e180} agentID="member2 IP"

anp member agent

I1011 03:13:44.931883       1 options.go:83] AgentCert set to "/var/certs/agent/proxy-agent.crt".
I1011 03:13:44.931951       1 options.go:84] AgentKey set to "/var/certs/agent/proxy-agent.key".
I1011 03:13:44.931959       1 options.go:85] CACert set to "/var/certs/agent/ca.crt".
I1011 03:13:44.931964       1 options.go:86] ProxyServerHost set to "proxy IP".
I1011 03:13:44.931969       1 options.go:87] ProxyServerPort set to 8091.
I1011 03:13:44.931976       1 options.go:88] ALPNProtos set to [].
I1011 03:13:44.931993       1 options.go:89] HealthServerPort set to 8093.
I1011 03:13:44.931998       1 options.go:90] AdminServerPort set to 8094.
I1011 03:13:44.932004       1 options.go:91] EnableProfiling set to false.
I1011 03:13:44.932009       1 options.go:92] EnableContentionProfiling set to false.
I1011 03:13:44.932014       1 options.go:93] AgentID set to 8ee90d92-389c-464e-a02e-d565db7347f7.
I1011 03:13:44.932019       1 options.go:94] SyncInterval set to 1s.
I1011 03:13:44.932026       1 options.go:95] ProbeInterval set to 1s.
I1011 03:13:44.932031       1 options.go:96] SyncIntervalCap set to 10s.
I1011 03:13:44.932035       1 options.go:97] ServiceAccountTokenPath set to "".
I1011 03:13:44.932049       1 options.go:98] AgentIdentifiers set to host=xxxxx.
I1011 03:13:44.942673       1 client.go:243] "Connect to" server="a93a9fe0-addc-42cf-a8f7-446682d670f8"
I1011 03:13:44.942703       1 clientset.go:196] "sync added client connecting to proxy server" serverID="a93a9fe0-addc-42cf-a8f7-446682d670f8"
I1011 03:13:44.942730       1 client.go:345] "Start serving" serverID="a93a9fe0-addc-42cf-a8f7-446682d670f8"

我知道怎么回事了用了自定义证书，会报错。。。。。我之前以为是缺crd，现在已经去掉了

kubectl get cluster --kubeconfig karmada-apiserver.config

Error from server (NotFound): Unable to list "cluster.karmada.io/v1alpha1, Resource=clusters": the server could not find the requested resource (get clusters.cluster.karmada.io)

如果用的auto

certs:
  auto:
    hosts: [
      "kubernetes.default.svc",
      "*.etcd.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}",
      "*.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}",
      "*.{{ .Release.Namespace }}.svc",
      "localhost",
      "127.0.0.1",
      "xxxxxxx"
    ]

kubectl get cluster --kubeconfig karmada-apiserver.config
No resources found

之前成功是因为我先安装的auto，然后卸载再安装custom的所以没有报错，后来我清除了本地机器的/var/lib/karmada-system目录，再安装custom就有问题了

生成证书过程

openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

生成命令

sed -e "s/{{IP}}/${HOST_IP}/g" $SSL_CONF > openssl.conf

openssl req -newkey rsa:2048 -nodes -keyout $SSL_DIR/ca.key -x509 -days $valid_day -out $SSL_DIR/ca.crt -subj "/C=xx/ST=x/L=x/O=x/OU=x/CN=ca/emailAddress=x/"
openssl genrsa -out $SSL_DIR/karmada.key 2048
openssl req -new -key $SSL_DIR/karmada.key -out $SSL_DIR/karmada.csr -config openssl.conf
openssl x509 -req -days $valid_day -in $SSL_DIR/karmada.csr -CA $SSL_DIR/ca.crt -CAkey $SSL_DIR/ca.key -CAcreateserial -out $SSL_DIR/karmada.crt -extensions req_ext -extfile openssl.conf

openssl genrsa -out $SSL_DIR/frontProxyKey.key 2048
openssl req -new -key $SSL_DIR/frontProxyKey.key -out $SSL_DIR/frontProxyKey.csr -config openssl.conf
openssl x509 -req -days $valid_day -in $SSL_DIR/frontProxyKey.csr -CA $SSL_DIR/ca.crt -CAkey $SSL_DIR/ca.key -CAcreateserial -out $SSL_DIR/frontProxyKey.crt -extensions req_ext -extfile openssl.conf

openssl配置

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
CN = system:admin
O = system:masters

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
DNS.1 = kubernetes.default.svc
DNS.2 = *.etcd.karmada-system.svc.cluster.local
DNS.3 = *.karmada-system.svc.cluster.local
DNS.4 = *.karmada-system.svc
DNS.5 = localhost
IP.2 = {{IP}}

helm 安装

helm install karmada -n karmada-system --create-namespace $PACKAGE_NAME -f karmada.yaml \
  --set-file certs.custom.caCrt=$SSL_DIR/ca.crt \
  --set-file certs.custom.crt=$SSL_DIR/karmada.crt \
  --set-file certs.custom.key=$SSL_DIR/karmada.key \
  --set-file certs.custom.frontProxyCaCrt=$SSL_DIR/ca.crt \
  --set-file certs.custom.frontProxyCrt=$SSL_DIR/frontProxyKey.crt \
  --set-file certs.custom.frontProxyKey=$SSL_DIR/frontProxyKey.key \
  --set certs.mode=custom

使用自定义证书下

改用最新的https://github.com/karmada-io/karmada/tree/release-1.3/charts 还是有问题，member注册失败的已经解决了

karmadactl get pod --kubeconfig karmada-apiserver.config --v=6

I1013 11:03:23.329736   12956 loader.go:372] Config loaded from file:  karmada-apiserver.config
I1013 11:03:23.329838   12956 loader.go:372] Config loaded from file:  karmada-apiserver.config
I1013 11:03:33.342750   12956 round_trippers.go:553] GET https://172.xxx:5443/apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api 503 Service Unavailable in 10012 milliseconds
I1013 11:03:33.342765   12956 round_trippers.go:553] GET https://172.xxx5443/apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api 503 Service Unavailable in 10012 milliseconds
Error: [cluster(member2) is inaccessible, please check authorization or network, cluster(member1) is inaccessible, please check authorization or network]

proxy api也用不了 apply 了这个

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  - member2
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-proxy-clusterrole
subjects:
  - kind: User
    name: "system:admin"

kubectl --kubeconfig karmada-apiserver.config get --raw /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes

Error from server (ServiceUnavailable): error trying to reach service: net/http: TLS handshake timeout

改用最新的https://github.com/karmada-io/karmada/tree/release-1.3/charts 还是有问题，member注册失败的已经解决了

使用1.3的chart过程中，做过什么改动吗？

有问题指得是下面遇到的关于agg-api-server的问题， member注册失败的已经解决了（对比1.2来说的，不再报错cluster.karmada.io/v1alpha1 not found 之类的，至于为什么不知道，我只是更新了镜像用了新chart）

不过1.3的chart确实有问题，刚开始连部署都失败。。。，之后我会提PR 主要是自定义证书要传ca.key, post-install-job 有些yaml错位

不过1.3的chart确实有问题，刚开始连部署都失败。。。，之后我会提PR

奇怪，目前chart安装在CI中是有覆盖的。欢迎提交issue，把报错信息贴出来。

主要是自定义证书要传ca.key, post-install-job 有些yaml错位

这个是怎么自定义的，是在chart安装时使用配置参数？

@RainbowMango 大概复现步骤 karmada-1.3.0.tgz 改动修复了1.3 chart 的bug https://github.com/karmada-io/karmada/issues/2628#issuecomment-1276978720 这里提了

然后就是把proxy server agent 塞了进去而已，没什么大改动

证书生成过程参考https://github.com/karmada-io/karmada/issues/2628#issuecomment-1274419201

#!/bin/sh
set -e

SSL_DIR=ssl
valid_day=36500
rm -rf $SSL_DIR
mkdir -p $SSL_DIR

ANP_SSL_DIR=anp
rm -rf $ANP_SSL_DIR
mkdir -p $ANP_SSL_DIR

PACKAGE_NAME=karmada-0.0.4.tgz
SSL_CONF=openssl_host.conf
AGENT_SSL_CONF=openssl_agent.conf
# Usage:
#   curl ... | HOSTS=... sh -
#       or
#   HOSTS=... ./install.sh
#
# Environment variables:
#   - HOST_IP
#     host集群IP
#   - PROXY_SERVER_IP
#     ANP proxy server ip

# 生成证书
sed -e "s/{{IP}}/${HOST_IP}/g" $SSL_CONF > openssl.conf

openssl req -newkey rsa:2048 -nodes -keyout $SSL_DIR/ca.key -x509 -days $valid_day -out $SSL_DIR/ca.crt -subj "/C=xx/ST=x/L=x/O=x/OU=x/CN=ca/emailAddress=x/"
openssl genrsa -out $SSL_DIR/karmada.key 2048
openssl req -new -key $SSL_DIR/karmada.key -out $SSL_DIR/karmada.csr -config openssl.conf
openssl x509 -req -days $valid_day -in $SSL_DIR/karmada.csr -CA $SSL_DIR/ca.crt -CAkey $SSL_DIR/ca.key -CAcreateserial -out $SSL_DIR/karmada.crt -extensions req_ext -extfile openssl.conf

openssl genrsa -out $SSL_DIR/frontProxyKey.key 2048
openssl req -new -key $SSL_DIR/frontProxyKey.key -out $SSL_DIR/frontProxyKey.csr -config openssl.conf
openssl x509 -req -days $valid_day -in $SSL_DIR/frontProxyKey.csr -CA $SSL_DIR/ca.crt -CAkey $SSL_DIR/ca.key -CAcreateserial -out $SSL_DIR/frontProxyKey.crt -extensions req_ext -extfile openssl.conf

# ANP proxy server
sed -e "s/{{IP}}/${PROXY_SERVER_IP}/g" $AGENT_SSL_CONF > openssl.conf

openssl req -newkey rsa:2048 -nodes -keyout ${ANP_SSL_DIR}/ca.key -x509 -days $valid_day -out ${ANP_SSL_DIR}/ca.crt -subj "/C=xx/ST=x/L=x/O=x/OU=x/CN=ca/emailAddress=x/"
openssl genrsa -out ${ANP_SSL_DIR}/proxy-frontend.key 2048
openssl req -new -key ${ANP_SSL_DIR}/proxy-frontend.key -out ${ANP_SSL_DIR}/proxy-frontend.csr -config openssl.conf
openssl x509 -req -days $valid_day -in ${ANP_SSL_DIR}/proxy-frontend.csr -CA ${ANP_SSL_DIR}/ca.crt -CAkey ${ANP_SSL_DIR}/ca.key -CAcreateserial -out ${ANP_SSL_DIR}/proxy-frontend.crt -extensions req_ext -extfile openssl.conf

openssl genrsa -out ${ANP_SSL_DIR}/agent.key 2048
openssl req -new -key ${ANP_SSL_DIR}/agent.key -out ${ANP_SSL_DIR}/agent.csr -config openssl.conf
openssl x509 -req -days $valid_day -in ${ANP_SSL_DIR}/agent.csr -CA ${ANP_SSL_DIR}/ca.crt -CAkey ${ANP_SSL_DIR}/ca.key -CAcreateserial -out ${ANP_SSL_DIR}/agent.crt -extensions req_ext -extfile openssl.conf

openssl_host.conf

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
CN = system:admin
O = system:masters

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
DNS.1 = kubernetes.default.svc
DNS.2 = *.etcd.karmada-system.svc.cluster.local
DNS.3 = *.karmada-system.svc.cluster.local
DNS.4 = *.karmada-system.svc
DNS.5 = localhost
IP.2 = {{IP}}

openssl_agent.conf

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
CN = proxy
O = system:nodes

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
DNS.1 = kubernetes
DNS.2 = localhost
IP.2 = {{IP}}

集群都是v1.19.6 ,

host.yaml

global:
  imageRegistry: "harbor.uniin.cn:20000"

certs:
  custom:
    caCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJAKqullP9pWqpMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEArKZbozzP+bl3G3lGhSpyCLRtaGuaTRDEaCJC+OFbQvKd8cvKfOsEvh0e
      ckdyfNyLdwRZnbwbLEQaC7DYnUcNx3shyX33wi1BiXIyLiJFv5cdJc2xoaVpPxas
      /A0svXP6X41p0R5138jp37+XRHkU/K8lTL7WQBseeWjd3WKz486WDUXWBBSpnC05
      wHTD/sq4sTMjhY3QEiQPqop5iPlknx/N720qekIpZTKw8E2jJrYG2xduoekX0+9A
      enqhfnIRgr9tDMYCKNM465FDZk0FcqzQeiRq+YcPAhCBJh7j8/Z4J/nnwsL9ZO8A
      xDuvRwMgINB42XdVTuSzY1gimD9oywIDAQABo1AwTjAdBgNVHQ4EFgQUSf/MUfna
      +fSpzVNAIcnc9SeLYiswHwYDVR0jBBgwFoAUSf/MUfna+fSpzVNAIcnc9SeLYisw
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAgq8NMsoFcgJmD+wxiP7e
      Bz76j3HAVadm50Ve5lT4IxRHhuN+xS2VU0fOxHnHKjKWDl+tkoYg0dwlDCnkg+GH
      umH5Gx617nix/QD2u6fgMfms7VqTVAPJddiMcx1v71j6vNjRv73LFNyV8OmTc9fx
      tWlQEtrfwtF5GiDiqMQ2jNmnbcAtMvlvBAg81Wc8yBJbjirjJCBHV7AYWuR9IbIf
      I747Nz6A/2ZcuqlaCDZIjYc3cMpvIj2BOSWN++oH1V2KYNiicz/yW5QOYL/HCfaE
      9kC1HHcRa2i5E44OUoU5oy390MGoiSJVa7wPQFxkrlRPCgsKxzEa3FiJxi5zFIEw
      Tg==
      -----END CERTIFICATE-----
    caKey: |
      -----BEGIN PRIVATE KEY-----
      MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCsplujPM/5uXcb
      eUaFKnIItG1oa5pNEMRoIkL44VtC8p3xy8p86wS+HR5yR3J83It3BFmdvBssRBoL
      sNidRw3HeyHJfffCLUGJcjIuIkW/lx0lzbGhpWk/Fqz8DSy9c/pfjWnRHnXfyOnf
      v5dEeRT8ryVMvtZAGx55aN3dYrPjzpYNRdYEFKmcLTnAdMP+yrixMyOFjdASJA+q
      inmI+WSfH83vbSp6QillMrDwTaMmtgbbF26h6RfT70B6eqF+chGCv20MxgIo0zjr
      kUNmTQVyrNB6JGr5hw8CEIEmHuPz9ngn+efCwv1k7wDEO69HAyAg0HjZd1VO5LNj
      WCKYP2jLAgMBAAECggEADjfA8idu5RzN3zhUg2DkdS0OmXvLAC+yTPUNWNuOHWxH
      b4Md0L217UYOnNCfcALlhU6ZZrD39+Q/1NiibglM7DGKrcIu5hG+BOifk0cFQWrb
      XqKazzoE+coPQs5+8bccVkQaA7ujb2NnRyLK/TuPa+Osia5pjGvvTYZCRxn6wHa/
      JEzBUvcA6/4K7rFdTGev7pxOetv5CzkfzSo/vqamu8bJEp/KA+vhQdgn5roDRFs9
      dan0E5fPHNkdfnHOd6wuK4ZATNX1PjBRWbCqAJzfcxZxr00du2G4M4eDO/boisNj
      cRVCqGVv8wGgfn/A159ZNK/cIKL2h+Px+iCbonoX4QKBgQDaIes+FsKsm8VliUvW
      N6msG0O93wfQyt+a0WX65i02lEOrwLtTMfQBD2F/pVWUn3ADA3wuUygABKG82CxD
      GdoOt8W0rVxaWJ49DX/MNI02X1m93ZfDMUqEvWvLFydGf+519DZ70BV9l7GV3FdI
      oCCfOWAquqnHvCottI6wclrL8QKBgQDKnyH5s4fIlmSPQmM5rsvXlopb3wsBi8D4
      CUkSBHHlKfOq9kFMOGuCtalD8Jya0ekBkegbMEKBmnNlxYkAQvrezf3CqWWq15R9
      aJ8eDZxTUpsuzJv5lPBtv6cYFNee5E9vIptB8ByewUJf2QifyrAvOLCKDZSHmnwz
      VZVsHrIsewKBgDTeO7HUumIzbuhpGmRVIV6BTS3bvH0ybneLgPgKibLMxIoAMGa9
      mLIZFDxnyhI3gmJ4mcRwiyVV+g1XJ4Mr5NnkWJqt2dBb4h5SA0NvCTrAaXPHHV1F
      uRXrAxE6D17a6uTTx9DCYjMKbRQxT0Ze6QkuUsyIaVXrmR1ywZKqZK9BAoGBAJkj
      hI4wviNAZ2hPCKeEENQ0FwYwoSTd905/1hYj/eDxwBQqYle1HKyBmBidNf9dPvnM
      yNwjsWA615THoy/rb6gNkAkb9pcUsEnIVNpl9wySlm7cfdY4Vy7bdvKgbgbND6wz
      P5yqWTczYXMHmZkLm/VUpzjv4p5KiCxUNsasvIUZAoGBAMGRuMNz2L6s1nFDfdeT
      8AB0yDuWRntJd39ysiKUOyF1OFkBOAYNOgTr1c9O+UQGhJmQaOLRB3t8//NT6OlM
      zEHVx2REf+vtSH4EjPJICiG77XTp1cPsCtuIXJIIunzw2VBDFsJtLi+/B6hJhHmf
      GDtJN4Mmw3my8nym+Y8s5ePM
      -----END PRIVATE KEY-----
    crt: |
      -----BEGIN CERTIFICATE-----
      MIIDtTCCAp2gAwIBAgIJAOihA6M0rQ8UMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owMDEVMBMGA1UEAwwMc3lzdGVtOmFkbWlu
      MRcwFQYDVQQKDA5zeXN0ZW06bWFzdGVyczCCASIwDQYJKoZIhvcNAQEBBQADggEP
      ADCCAQoCggEBANVIoTBuxlv0K7GF9T6kX7vYrPTMIA5EBZGfBF8i3HlnWaJdTzMD
      myAYIDzqI1fDKqbmYQ01JrDmwIveSIeGYp5M8eW0FsHyTo1jMqFpUe6j5T6XqOWq
      M0uIoK0fCbXOqcysCUU0OXQo2MOjAEkD0SNLG0i3DSXLWmbQjEGdLitsy91iQaSD
      7WkFDILfwtictmGopeLGZS6cjG0DD9zA5NMftD6JeyDq7MMlJKipiJ6jC8kmSLRF
      cy7jnFR8muZOtZOpz1YnuBOJZf35qW+rrtNcYGsTLoN5u/Se3CxCeVv3aXdIK4AP
      o1pFBZNCQNWPATf5bxe0kQ1wZ0VDyQaJpQkCAwEAAaOBozCBoDCBnQYDVR0RBIGV
      MIGShwR/AAABghZrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjgicqLmV0Y2Qua2FybWFk
      YS1zeXN0ZW0uc3ZjLmNsdXN0ZXIubG9jYWyCIioua2FybWFkYS1zeXN0ZW0uc3Zj
      LmNsdXN0ZXIubG9jYWyCFCoua2FybWFkYS1zeXN0ZW0uc3Zjgglsb2NhbGhvc3SH
      BKwRBzkwDQYJKoZIhvcNAQELBQADggEBAKjk51B2rDqr/ApQEbVhU66OlA4x2cww
      InLJzXrq0Y1Y0aVojAxucckhXgumuq0PwPJC9x9OgCXgDZo+nqlM8wcmrsv0T82R
      TvA4FATEpB29BaDMZjt9x8T9oWAH37gkCX8Ze2S50Pvt/vCukYxGydiKjjiwihr8
      xtkRtYBnt8gBRuzt+pt7IgvQDrhocbfNco4w8PmE1wFaLt4rVu4OWckABYNCqYVD
      /fmezWDLuyAOhu1Gzvy4xh32Uo4MPLDPBSlU7osXtYGf/qnjgko1gKv75GQ1Roy1
      UTZqbw1WgzGrxWBR82czJqbD0mnFQlSMDCgDddJRTuvcZxcR5PwlVBI=
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEogIBAAKCAQEA1UihMG7GW/QrsYX1PqRfu9is9MwgDkQFkZ8EXyLceWdZol1P
      MwObIBggPOojV8MqpuZhDTUmsObAi95Ih4Zinkzx5bQWwfJOjWMyoWlR7qPlPpeo
      5aozS4igrR8Jtc6pzKwJRTQ5dCjYw6MASQPRI0sbSLcNJctaZtCMQZ0uK2zL3WJB
      pIPtaQUMgt/C2Jy2Yail4sZlLpyMbQMP3MDk0x+0Pol7IOrswyUkqKmInqMLySZI
      tEVzLuOcVHya5k61k6nPVie4E4ll/fmpb6uu01xgaxMug3m79J7cLEJ5W/dpd0gr
      gA+jWkUFk0JA1Y8BN/lvF7SRDXBnRUPJBomlCQIDAQABAoIBAHrQidED2MEWhxqA
      UsoxiNb4XA/Sgxrz3GkQMko+Hsrh0mGGbZaomukOO3IYOSXOQn885eHfG1p+ZZ2k
      8ArqPVt+X0SwQPKJfuY3mu4LYZyTT+yNFDq7mTIDiNJnrwW77VboMaZh/jt/UeLF
      t+JTilhSZXGnxrBeur29lSYrRmzF+i7X94MakM/LhPtttc9+SwVnw8EWWXsQ6KPg
      3X8c1eiSHemlogOxdTaGp+RtekUu9CEQmTUFRgVUGMcPRYW2fQlhe5WRy4NDZJQD
      l7P/k1LZbxD1131e7QnlK5aOL0ehjt8eDSYFUsnM3BRRDoZ2jZfgcoW0+TaOJ00h
      soHE7iECgYEA7gxUdvKYqFeM3tFtjrhybmPAVqXSec4Vs8vMqHod11GfqZeYjRCh
      RFcsfqped9vwdrquTTU5YAx3m1FqII/ldmOS8r96y+4WKBCqgZ+PwMj2yHXvoeK5
      vf60Z+lIYEWyIiFlAorVuDVP5k6MYNIVe9xLdGqCF5RZgEJ5E+/MmcUCgYEA5V40
      zwd8q5tdT2NA5FQgfun8RcoefVLcpbhcjyMuRdBaS1ed3C/Ea+z0rxbvglPKcCdO
      Kpc4gBa55IbLyecUERLgI3TltR++68SUSE8oUkJC9P+6jKyJfV1nvMfCzJGvnXuZ
      FMT25BTv31hgDn+GrkYic33dUgJW/rRCCJlFxnUCgYAz2cWwHwl3aZKKmi117+El
      AF9NwFLHK0d44URT0j2bv/C6HQYie+PUofRUJsHq86I+Ck2bDaPQswZ+c05dh1pg
      3jGM5H5X2yVCVtb84weJ43w1j9LBfUQeCVO8aojUR3oUixeYZStGM36S2Lsk/72k
      PAjHc8egYTTZhiEfYZU8TQKBgFkmqb4Cefm1lKojuPhGnq7VwaFas0kg6PRc2VUl
      ZnnqRv5b2tbXljAg1OYtMwDkEp5PGLnDN0nKC40Au7sJXypXvNjrOdGRukKnkL8e
      TVzTtnTF5TuCYJpisMyWlGexBKJghRqmiBQU8ztLfWKFPNMGHaPoOxsCH9A2Pihb
      ctx1AoGAGN4WilL6wxw6hpHEsK7NfMDnr8I1x4ieVXcRyhukZmWgRA0HsfeBPeOm
      to4kKPPtJQ6zNKVAWxJTF3XKX3Z4oAnsyi2cUTw2+KBcMApnIZF/vNkkPy/nxgap
      uz1sboIKT01Tk0nSNKL+i6MzTGbfDnCqhdK9RYYaq7yyxI38uBI=
      -----END RSA PRIVATE KEY-----
    frontProxyCaCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJAKqullP9pWqpMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEArKZbozzP+bl3G3lGhSpyCLRtaGuaTRDEaCJC+OFbQvKd8cvKfOsEvh0e
      ckdyfNyLdwRZnbwbLEQaC7DYnUcNx3shyX33wi1BiXIyLiJFv5cdJc2xoaVpPxas
      /A0svXP6X41p0R5138jp37+XRHkU/K8lTL7WQBseeWjd3WKz486WDUXWBBSpnC05
      wHTD/sq4sTMjhY3QEiQPqop5iPlknx/N720qekIpZTKw8E2jJrYG2xduoekX0+9A
      enqhfnIRgr9tDMYCKNM465FDZk0FcqzQeiRq+YcPAhCBJh7j8/Z4J/nnwsL9ZO8A
      xDuvRwMgINB42XdVTuSzY1gimD9oywIDAQABo1AwTjAdBgNVHQ4EFgQUSf/MUfna
      +fSpzVNAIcnc9SeLYiswHwYDVR0jBBgwFoAUSf/MUfna+fSpzVNAIcnc9SeLYisw
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAgq8NMsoFcgJmD+wxiP7e
      Bz76j3HAVadm50Ve5lT4IxRHhuN+xS2VU0fOxHnHKjKWDl+tkoYg0dwlDCnkg+GH
      umH5Gx617nix/QD2u6fgMfms7VqTVAPJddiMcx1v71j6vNjRv73LFNyV8OmTc9fx
      tWlQEtrfwtF5GiDiqMQ2jNmnbcAtMvlvBAg81Wc8yBJbjirjJCBHV7AYWuR9IbIf
      I747Nz6A/2ZcuqlaCDZIjYc3cMpvIj2BOSWN++oH1V2KYNiicz/yW5QOYL/HCfaE
      9kC1HHcRa2i5E44OUoU5oy390MGoiSJVa7wPQFxkrlRPCgsKxzEa3FiJxi5zFIEw
      Tg==
      -----END CERTIFICATE-----
    frontProxyCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDtTCCAp2gAwIBAgIJAOihA6M0rQ8VMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owMDEVMBMGA1UEAwwMc3lzdGVtOmFkbWlu
      MRcwFQYDVQQKDA5zeXN0ZW06bWFzdGVyczCCASIwDQYJKoZIhvcNAQEBBQADggEP
      ADCCAQoCggEBAKer5NtUHhNdDo8VHMf3G/IxTB9W7PAYjtzHyC3nd1Zusfl4MplB
      bOpmQ+hcOml2tBZQNhZNuyUpNyhq/mAAEEVbz3UR0y3MaTfPBffPwH1q26ZfRBdd
      LAWYPIBCBFmmYv6Jriup8NMRFZYbKUMgCoouMW3k1BVfVuy1DL4k6b8HlV3gRhI2
      hSqK8Lk9uN2YWhHpp5VRWhCkMOIqvjwfgkGKcNU3CSZJDVjzgHU5Cdvf87ftYxMe
      aem54D6AXIyoVEamhswJ+oAfNCBof8FrVX079txsZX7764XcfI2Zfn67ndk84mzH
      6I0KwvrnXKoXYERQwE5bmxxIHX6+hp7gx/sCAwEAAaOBozCBoDCBnQYDVR0RBIGV
      MIGShwR/AAABghZrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjgicqLmV0Y2Qua2FybWFk
      YS1zeXN0ZW0uc3ZjLmNsdXN0ZXIubG9jYWyCIioua2FybWFkYS1zeXN0ZW0uc3Zj
      LmNsdXN0ZXIubG9jYWyCFCoua2FybWFkYS1zeXN0ZW0uc3Zjgglsb2NhbGhvc3SH
      BKwRBzkwDQYJKoZIhvcNAQELBQADggEBACb7/j5ZKsgSkgsht1KhUWx54cj2Gwmv
      cjU5s/mUWCcPJ2Ub5ibfxWQX/iOTgP9ETeoN4vieuJ1b+T322N9x8OZ6hTuGomJa
      5UPeJKSlOu6VOyLCfXXlVznoOqkkyzpejOfHzTJT/fLhCCUQHFtYL6tTuBaXrVtX
      50tXFN8FfNb8XDOM+3BJcIMmMn88qp6siFpvyi9Y4BhgumurE4uJbdTQL0Fagbol
      HD54aR42PJvjw6POQP0Ua788CDOh5ikpDP37lA1bVgo/thX5ctlBprAoxFzCvSqn
      41e74hFNLsacavB1gtog03VYBsuR7lKZujn00vyoZc8dfC4yRmXBrLs=
      -----END CERTIFICATE-----
    frontProxyKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAp6vk21QeE10OjxUcx/cb8jFMH1bs8BiO3MfILed3Vm6x+Xgy
      mUFs6mZD6Fw6aXa0FlA2Fk27JSk3KGr+YAAQRVvPdRHTLcxpN88F98/AfWrbpl9E
      F10sBZg8gEIEWaZi/omuK6nw0xEVlhspQyAKii4xbeTUFV9W7LUMviTpvweVXeBG
      EjaFKorwuT243ZhaEemnlVFaEKQw4iq+PB+CQYpw1TcJJkkNWPOAdTkJ29/zt+1j
      Ex5p6bngPoBcjKhURqaGzAn6gB80IGh/wWtVfTv23Gxlfvvrhdx8jZl+frud2Tzi
      bMfojQrC+udcqhdgRFDATlubHEgdfr6GnuDH+wIDAQABAoIBAAu88OyZ2vEoFpjV
      tYoaNo6h/N5Ey5mTXoz+P3lXNveKEFOSopgwCFvwhNiDfGNduAXH38F3954GYKIn
      u18lIf55CvD7KnmOUd9iUtyiYqtlC7N9/9pS334UxkeEV6GfEFcZdGQCJaXhEE3N
      4PmeCJ7b1QIIfwBwCDnSAu5czzBcToJb3KSj3c6I0GLvFJVTTYp94WuCfZgMmBy3
      sG1h5vXnRPIa+htvS8A5snJ5rcai6s6CG1/veaKKhbRludqeeWpZQEDLo+4zSR6K
      S9bFXI24RUSe+LX+k8q97XlG+SycUQXPE1NrkY61BpPawYR8wK5EYqw7k8hRt/s6
      ZP1QoiECgYEA2RqOjlpxfcZ+X+kvoikWZfXbZ/nkbNIERUpHdz08cFRM9L0snrVZ
      G749avZtX5+3OaBe7IOksET3XxQ5GPXZbGkyLgRtUwUqFIWpyCAsAWFr429PlHIC
      6zXx+U6mVHlilgvCIJ1hL0XV8xaRZwSqiEvWnMQTGdSB4ytTpkm0wZUCgYEAxbYh
      V0V3qxQ5Q5pLt2FDhfnwkanwGfbbufjDhTTE64x5XTeD0d3/1fSTkD2d5ZYQ/PhC
      P/znOWtkFMEv7THoWPoY6qHmCyTZfFmbGOGuvQ9Eo1WqBat6Wi879yBZBo6G6qtT
      aYKvbe+SQeS1L36vvFmBc6GXvKeppIMMOzAIH08CgYEAyfs3HaFvbQC6miisNEXd
      nMZ7deS7a7KIXp6YlorWK/uCDT/iv1Ugrfrz0Quq/gftwyM8rPZcxdGkA0UhSeOQ
      gJfuI3ni7BYl7Cnep12nEzG72kC43PalWIc7tCJEBNtZqRcPaKCCxiRGxH3+kMV6
      kotcA0U4MBPRM7QfgtmX3XECgYAmKM3skyrSqTeYNaqRfUNUE3Z8FsmRqVSlYz8R
      carkL5upAzL9CZT8TZVBjLLyAfHLcXsP8JcMPQ729oZyiGRo7/RpLNBCrF0RpBXM
      Spq4QL8lqz4ClCMsTtuLCgJEwrOFjYWu4ptLY6bd8u+JeDEg8EGM+M5Zih6y8twk
      fFPE/wKBgG4MIu7ZIAO/J4EdjAMACKrhtO/te6RhBlz3hORnSxjFpASi3vPpuPZQ
      ElY1HhXtFWA2q1kc73V8X2LtzRTPhmPvGRFqOtlrn0UXd7MlS5/MV+tHnKqLtimS
      1kPF55MkCsOLDCRIGRZ7A0ihf+RCG/W1HXWUYMPS/TQpsFDx00cv
      -----END RSA PRIVATE KEY-----
  mode: custom
proxyServer:
  install: true
  cert:
    caCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJANIhggLSSlYtMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEA4lN3G30VKwtnpqLZiB/0NK+pNtPdPX+/iz21D7EuF8kgpryJ9PecFW7I
      OVMH82GpWqpjPbbVSCVph67lq0kKZ4nW1YsGwkmY9qBZCwGpl/9tj7Txx7xJPaGF
      +uILrfqlDPMEdUBdc5ceZBlsz8xdHWdfFTdXCEwtKQlaSAzUbPINHUQox/A2pGdj
      8p6omiEWlNUPVruexNzdHWJjtKIhh9L0iL0hBb6eNIO/rSqoZ64X9oqjRuQMwX9j
      UXoV3fOQ5yFo/z501DPBRMzI2KGKWT7TqzZqmVNiNpzvvQFeDJ2zFutUHKye3IPN
      0gpktPZ7W6ROW2bYOptLUN0GvP/V2wIDAQABo1AwTjAdBgNVHQ4EFgQUB/FrMnWn
      7Szrwz4coYkGQVH/ei0wHwYDVR0jBBgwFoAUB/FrMnWn7Szrwz4coYkGQVH/ei0w
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAvl2lJzozUfFlj2Plowvd
      LB64RXDR78kW0SKfBLSKbpLcgR3O9AkZ17Wgtbcr3EmV7Wmg0vl1JigxgtVVY6ns
      jjeuod13PKOC1O8+a2qfWtixv4Y0y2plAUSYvobDXOcqzfLEs/WudwNi7cDtBRth
      K81O2Wd1k4R7txLup+uESZfwWf2vXJoO8PvTQuyG1ihtdA4P/ZIr7P7u2qAbJzDX
      Mgy79ANhuiPt47g5UNbp25dVGVMFPYViUj1r5zeZQrZK+bC4lzQpQFdY5OXO7DbV
      /b/6uZyrFmLasLyppz0L0I6L0hFsDnbCmjY49GhQePlTz+mLaafbeW9HJiZseePB
      Cw==
      -----END CERTIFICATE-----
    crt: |
      -----BEGIN CERTIFICATE-----
      MIIDODCCAiCgAwIBAgIJALfJxEK+C6bNMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owJzEOMAwGA1UEAwwFcHJveHkxFTATBgNV
      BAoMDHN5c3RlbTpub2RlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
      AMiqvSN6d5ExxzvzgKLVfjc1V53wnFeJFWtne3PTJbsfZIcPSpe2+sLSMWbvXYi5
      I2HsgF3vqXEoLgj87Yd1zi0zra9ZoRx+njumdmgKCQRWOZpIG7rDYNcvBzNRidAv
      mDvMC88oFT39BS6UTfB7qgVWktqCosmpK8m/jujnreWdE7WkOJgjZIMZ32Vr21A9
      5ooT+dasKqJs2vPQ7atmXigjW4dloKixQ73qjHlWxDjxxyBpJuCTbceMoT5ZOttH
      WwLBh5Tz1kCImqniAYBG/782ZHo5GWe9M1XeHMyzFl9IMe0lvYcQR28+uGdS3nNf
      TaoNtyUqKXvBZkcCxHOeNzkCAwEAAaMwMC4wLAYDVR0RBCUwI4cEfwAAAYIKa3Vi
      ZXJuZXRlc4IJbG9jYWxob3N0hwSsEQc5MA0GCSqGSIb3DQEBCwUAA4IBAQCFkp0e
      TiaUhbL9g/xIjG3mx6vpkQ4ybWEmqZO0VVq4lHaRTc/OYlIRw5r38s5kOkE+bkT2
      ZzMgYWRk0Snw4+GjbkaaFpvayB1PE3Eza4yTEgyo4cqDH7sRUgaD7jK5/SV8Asub
      +k7X2NuJKbUVzAjl0KEEuyI9g4c8+mpXdjXIEmr+B1kiZoya5i8M9Nxd1OW5nncH
      dOV01P4PiXE8BOXKjtLkYjmFEq1hVAiXbgqyTeWXOP1zLGngicxkxMS22PEodY8k
      iwVnxBJV71R72h6pg5IlNWqZPaI+IeBagLKrIK16MCGMZ8HLtMr7YfTqa7t3KCHa
      PmxSrluCF40RhlGN
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEogIBAAKCAQEAyKq9I3p3kTHHO/OAotV+NzVXnfCcV4kVa2d7c9Mlux9khw9K
      l7b6wtIxZu9diLkjYeyAXe+pcSguCPzth3XOLTOtr1mhHH6eO6Z2aAoJBFY5mkgb
      usNg1y8HM1GJ0C+YO8wLzygVPf0FLpRN8HuqBVaS2oKiyakryb+O6Oet5Z0TtaQ4
      mCNkgxnfZWvbUD3mihP51qwqomza89Dtq2ZeKCNbh2WgqLFDveqMeVbEOPHHIGkm
      4JNtx4yhPlk620dbAsGHlPPWQIiaqeIBgEb/vzZkejkZZ70zVd4czLMWX0gx7SW9
      hxBHbz64Z1Lec19Nqg23JSope8FmRwLEc543OQIDAQABAoIBAF/Gnl/ApDD4wU8r
      +cm+fG99VkRcA0MEeuS4j4PI08lsQl70PXYML8Os70diGBKZyvmCGAK90PHha5d+
      oSB2aeHJOKzo6qzfAkcWPI2LvKIFRZkNsa6CckZCe/1SfaSQEuj3iutlBj5/9Wk1
      /704SP4jRBRnJat6wxuYAm3Hpohgh/x/nEHnlAItOgvLn/c/X7eB+tfJRyUUMRVY
      I9h/BLoyWZS5+rHc4NumzZWTDvj9KEsTBgaPEWyG/QKmwyZBURFHwJo9/38/J9ni
      IKT85d92bEJFuJ8xfFg5hxiXA13vCLy7vUkH3L/IIu+WtUMDWulwphzF+xpZY7OK
      ICA3SwECgYEA6mg+YNFXMLHICUvR08LJOmvUmd21mnvA4vPF9HGHWnwJBN6C3u7i
      dfaVUH9UXi+P4VVHp8VI62YPZhYuI2kQM+AgPq6PdeiKrk3zdVr5Gjlg2dyGsM3D
      4htVxpC6IFmdNSQLhI1vZar/WlDXVyyJcwxcr9WkyeHRdCgw6hGS/zUCgYEA2ybW
      YpE1kNczv+VwStXf3EAEj46Z3KMb4J0aWZjqKuF4lsP2Z0ak4PU7fkPbmex5lqJx
      43bH4/jLU/UzCCZcaaJ+C3hHGHoMh/rzLOO4ZeFD1VCvLT4ZCvZC8N1QVdWXSniq
      LnXatn3jrtGGnoFtxMYwtHFWTJyE+oatjQo/xHUCgYAIFyz//nm57oju+TT20CRN
      5ebioKph+J+BtdRBSaHYV6ugBWYS97/NPoOM0bhBv/aBWoYsnjyUE2O4rE9PXZL3
      l2bFgxTkhvAyb/JVPJyYcR6MpnXnqFJ4fWUKg3hJIY6xhNr46kDMC4jgvyWPlrPv
      yp9wWjlj1m16SuDVU2H1eQKBgEhsS0kDp+oX1FhEYJm6mGJXHaq7e4l4tcvC32Rg
      rnJBVPKuDA4RQIe/Q0v07OuvRHdNc8myBVkHbVpvm8/DraKTOohb1VL3cpC1lAtv
      rg6tJRentFyFOw1uWKz01omnLnpD6im0dPT+IXIyeASFzVrMDGh1VAm5aBDK1SvZ
      CT3JAoGAOXlJx8eFI6MSccU79MNTN3CoieW7rGAUdcGVEnfX3hTUVs6Wzl2Ogs/9
      W6Z9Jc4t2krlC6pKTx2XamCHOgWAu7ynsF8iqg3CsMjQFS75+6Z0A3q0YdGqSbpv
      VWxzJf2OHGCV6E6P+0VQat0CzQFD5+PWHXvPz5dwYTlq9ZkEnSM=
      -----END RSA PRIVATE KEY-----
    clusterCaCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJANIhggLSSlYtMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEA4lN3G30VKwtnpqLZiB/0NK+pNtPdPX+/iz21D7EuF8kgpryJ9PecFW7I
      OVMH82GpWqpjPbbVSCVph67lq0kKZ4nW1YsGwkmY9qBZCwGpl/9tj7Txx7xJPaGF
      +uILrfqlDPMEdUBdc5ceZBlsz8xdHWdfFTdXCEwtKQlaSAzUbPINHUQox/A2pGdj
      8p6omiEWlNUPVruexNzdHWJjtKIhh9L0iL0hBb6eNIO/rSqoZ64X9oqjRuQMwX9j
      UXoV3fOQ5yFo/z501DPBRMzI2KGKWT7TqzZqmVNiNpzvvQFeDJ2zFutUHKye3IPN
      0gpktPZ7W6ROW2bYOptLUN0GvP/V2wIDAQABo1AwTjAdBgNVHQ4EFgQUB/FrMnWn
      7Szrwz4coYkGQVH/ei0wHwYDVR0jBBgwFoAUB/FrMnWn7Szrwz4coYkGQVH/ei0w
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAvl2lJzozUfFlj2Plowvd
      LB64RXDR78kW0SKfBLSKbpLcgR3O9AkZ17Wgtbcr3EmV7Wmg0vl1JigxgtVVY6ns
      jjeuod13PKOC1O8+a2qfWtixv4Y0y2plAUSYvobDXOcqzfLEs/WudwNi7cDtBRth
      K81O2Wd1k4R7txLup+uESZfwWf2vXJoO8PvTQuyG1ihtdA4P/ZIr7P7u2qAbJzDX
      Mgy79ANhuiPt47g5UNbp25dVGVMFPYViUj1r5zeZQrZK+bC4lzQpQFdY5OXO7DbV
      /b/6uZyrFmLasLyppz0L0I6L0hFsDnbCmjY49GhQePlTz+mLaafbeW9HJiZseePB
      Cw==
      -----END CERTIFICATE-----
    clusterCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDODCCAiCgAwIBAgIJALfJxEK+C6bOMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDhaGA8yMTIyMDkxODA5MzgwOFowJzEOMAwGA1UEAwwFcHJveHkxFTATBgNV
      BAoMDHN5c3RlbTpub2RlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
      AMTXS3oG3QJEwNCZCwTyDFivlQD6W0lNEPSILk7d3gQLW+Fzgzgnfq7vLTBa4WeI
      8FdNRWG6JjVDrqBLC7kcQ2W//Me8n0qtPyKm8uIwvGtFZX+/0z5mtGrUzWgv6Wz8
      yWI/2tBcGkBj6aUncfWH/yXDzu24IW7mguhXsiUCv1TYqTB9S0SJPVAO6U4VlAtD
      2BY0WK0BoaL2GDPfeBlGb35zExwvTAqFv79WbyHZmpZ0aZ7LavK/EqlUCi/IuWMg
      blpp02kE1vHXwAhak0dBe5i4IoJVlEa5MFz8nxg5o8DL8sl6D+ZRB+BczUOnmVkY
      +r0YApfYBO9R2V+J2CJX6kMCAwEAAaMwMC4wLAYDVR0RBCUwI4cEfwAAAYIKa3Vi
      ZXJuZXRlc4IJbG9jYWxob3N0hwSsEQc5MA0GCSqGSIb3DQEBCwUAA4IBAQBIrH9N
      XYgh1yvd0D2YBh79wQ2GfTQlsegaHxpua8LADZZ0IwFNUN2wSaLZzg3i+pC4kubU
      ILws65pW48U0vHF5EQ6tZ/WGBM6u5k6Q9vkPRX9yT+OPDse5nGJMVLOmlwP65V6o
      mWFako5nvizVRDV5RYiY+KlgtMZpIUUe57yodPDysBcXBXMhQ4Sk4zwBwy5+q2uO
      Ux/Y4O1StxQ5QzTLAgr1vI+VC+yf3ho8Clf5cDbKy6TNwqRLiI0Q7DVU906YZgl/
      kI7UhhvZTCVFwteAVTZFWiwWJKsImypodP1zNc8413hI65pygWjHBYDXX9YR8md9
      FkzRNd1l0T21eP+s
      -----END CERTIFICATE-----
    clusterKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpAIBAAKCAQEAxNdLegbdAkTA0JkLBPIMWK+VAPpbSU0Q9IguTt3eBAtb4XOD
      OCd+ru8tMFrhZ4jwV01FYbomNUOuoEsLuRxDZb/8x7yfSq0/Iqby4jC8a0Vlf7/T
      Pma0atTNaC/pbPzJYj/a0FwaQGPppSdx9Yf/JcPO7bghbuaC6FeyJQK/VNipMH1L
      RIk9UA7pThWUC0PYFjRYrQGhovYYM994GUZvfnMTHC9MCoW/v1ZvIdmalnRpnstq
      8r8SqVQKL8i5YyBuWmnTaQTW8dfACFqTR0F7mLgiglWURrkwXPyfGDmjwMvyyXoP
      5lEH4FzNQ6eZWRj6vRgCl9gE71HZX4nYIlfqQwIDAQABAoIBAQDDahFuNX4XkHXz
      INyFrl5/dOByNhmGLdIVUVTXyv9r+HLg4TJ1/w+PT7cG+FIp3RLtOLMWkBz6aBTm
      4VnkDaEDVJP5HtLs6g1xs1gRId0Mibp4FtH6cy/NlzHMqzQLlPHAbjAnBiixn/8G
      jKBf5wFwVnIKIfBfDs0LKHSctZuI+EkrJ2AQQ5lCDQ3J8GARW5bD1GJ9IZh9m1W8
      eCcIZqKCQhUu9LdNJZKLj1PEfkkEerQv/VEveZVK+udVBavxqeK8aObQuurNwdyG
      xTdW59tzLZ8760u5z6CSjSHlXi8hSPWIdzWKF3In78Yt50soRYZ4svXD2q3aqBlW
      B1U6QF2RAoGBAObku8cVM7YJzVwWPc52PoPnyfpxtbhGWbo8UASDrSAYd1uHEZFW
      tHOd1Jropq5oVDSShaV7r9RABR0cRp5UF4asQCIgwxKqE+1I6O1i0YlN8HTP398W
      +K+rzpsLEGI9bigoQlUqzron4sE2iZOHBmFfA2OMQSqIsY5NRAL7M3FFAoGBANo+
      qLkPmRrf5bdW1wAluZnX+y4KjZNSGROH7uDFNcw8O9LWivi2xeVUB4BmLCQc7KLk
      931YvFQe8QjrIljJ/z9mSPf3gILsE2PJXEQxa0lO3Ck1nAJ1uLPdwNd2GXnMHGS5
      GzfwCJITqPLGpA7vYi/8vPRCji6jQpudRoky7LHnAoGARc9sswHHE8gc7jXPVChO
      OIWRfQZ6Yg/rOzLXIrZJVwXuuJ67Ag2MG9ThnkCvfNYQd6DBhcpC3FLK+QU+mMpm
      HpLgBTGQqYkA42KDAps5LWGCTV84mfQRFtREcH5E4t8d+khJ3gRGYuSCW9bIe/F8
      TV5z4NpcLacMwDC+WoRnBl0CgYEAvJ3qtAYHDHL01b73kekIvy0qqF/2e5w4j+ir
      Sxclp51nFpDVCp7r6Yb19MJY490bpqy8OFi4avtS1aJhzNrG/sQnzCbbqkrddyBP
      3Z/YWUMOg48Rhg630ruULqKi4eDFqUdRfsa0oYrcOPX0U/QPp+mGZJQgza3LozIK
      BAk/B4ECgYAhLMj7zy9pTlsvT38lKdSyDVGCg2WQIDJV1owL78eJ3LruPoGSYl/I
      jYnDEhGCzg9AlqMnL8PhWD+6zwrWVvxBR3LT5qy5v26BR+KNum9AU7Pi/2NlyicU
      JQtevMj8wf7/rYvnE9ryiiyH0jEm5SRKFqHTkMlWcMQit5ggQHlJnw==
      -----END RSA PRIVATE KEY-----

member.yaml

---
global:
  imageRegistry: "harbor.uniin.cn:20000"

installMode: agent
agent:
  clusterName: member1
  clusterEndpoint: "https://172.xxxx:6443"
  kubeconfig:
    server: "https://172.1xxx:5443"
    caCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJAKqullP9pWqpMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEArKZbozzP+bl3G3lGhSpyCLRtaGuaTRDEaCJC+OFbQvKd8cvKfOsEvh0e
      ckdyfNyLdwRZnbwbLEQaC7DYnUcNx3shyX33wi1BiXIyLiJFv5cdJc2xoaVpPxas
      /A0svXP6X41p0R5138jp37+XRHkU/K8lTL7WQBseeWjd3WKz486WDUXWBBSpnC05
      wHTD/sq4sTMjhY3QEiQPqop5iPlknx/N720qekIpZTKw8E2jJrYG2xduoekX0+9A
      enqhfnIRgr9tDMYCKNM465FDZk0FcqzQeiRq+YcPAhCBJh7j8/Z4J/nnwsL9ZO8A
      xDuvRwMgINB42XdVTuSzY1gimD9oywIDAQABo1AwTjAdBgNVHQ4EFgQUSf/MUfna
      +fSpzVNAIcnc9SeLYiswHwYDVR0jBBgwFoAUSf/MUfna+fSpzVNAIcnc9SeLYisw
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAgq8NMsoFcgJmD+wxiP7e
      Bz76j3HAVadm50Ve5lT4IxRHhuN+xS2VU0fOxHnHKjKWDl+tkoYg0dwlDCnkg+GH
      umH5Gx617nix/QD2u6fgMfms7VqTVAPJddiMcx1v71j6vNjRv73LFNyV8OmTc9fx
      tWlQEtrfwtF5GiDiqMQ2jNmnbcAtMvlvBAg81Wc8yBJbjirjJCBHV7AYWuR9IbIf
      I747Nz6A/2ZcuqlaCDZIjYc3cMpvIj2BOSWN++oH1V2KYNiicz/yW5QOYL/HCfaE
      9kC1HHcRa2i5E44OUoU5oy390MGoiSJVa7wPQFxkrlRPCgsKxzEa3FiJxi5zFIEw
      Tg==
      -----END CERTIFICATE-----
    crt: |
      -----BEGIN CERTIFICATE-----
      MIIDtTCCAp2gAwIBAgIJAOihA6M0rQ8UMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owMDEVMBMGA1UEAwwMc3lzdGVtOmFkbWlu
      MRcwFQYDVQQKDA5zeXN0ZW06bWFzdGVyczCCASIwDQYJKoZIhvcNAQEBBQADggEP
      ADCCAQoCggEBANVIoTBuxlv0K7GF9T6kX7vYrPTMIA5EBZGfBF8i3HlnWaJdTzMD
      myAYIDzqI1fDKqbmYQ01JrDmwIveSIeGYp5M8eW0FsHyTo1jMqFpUe6j5T6XqOWq
      M0uIoK0fCbXOqcysCUU0OXQo2MOjAEkD0SNLG0i3DSXLWmbQjEGdLitsy91iQaSD
      7WkFDILfwtictmGopeLGZS6cjG0DD9zA5NMftD6JeyDq7MMlJKipiJ6jC8kmSLRF
      cy7jnFR8muZOtZOpz1YnuBOJZf35qW+rrtNcYGsTLoN5u/Se3CxCeVv3aXdIK4AP
      o1pFBZNCQNWPATf5bxe0kQ1wZ0VDyQaJpQkCAwEAAaOBozCBoDCBnQYDVR0RBIGV
      MIGShwR/AAABghZrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjgicqLmV0Y2Qua2FybWFk
      YS1zeXN0ZW0uc3ZjLmNsdXN0ZXIubG9jYWyCIioua2FybWFkYS1zeXN0ZW0uc3Zj
      LmNsdXN0ZXIubG9jYWyCFCoua2FybWFkYS1zeXN0ZW0uc3Zjgglsb2NhbGhvc3SH
      BKwRBzkwDQYJKoZIhvcNAQELBQADggEBAKjk51B2rDqr/ApQEbVhU66OlA4x2cww
      InLJzXrq0Y1Y0aVojAxucckhXgumuq0PwPJC9x9OgCXgDZo+nqlM8wcmrsv0T82R
      TvA4FATEpB29BaDMZjt9x8T9oWAH37gkCX8Ze2S50Pvt/vCukYxGydiKjjiwihr8
      xtkRtYBnt8gBRuzt+pt7IgvQDrhocbfNco4w8PmE1wFaLt4rVu4OWckABYNCqYVD
      /fmezWDLuyAOhu1Gzvy4xh32Uo4MPLDPBSlU7osXtYGf/qnjgko1gKv75GQ1Roy1
      UTZqbw1WgzGrxWBR82czJqbD0mnFQlSMDCgDddJRTuvcZxcR5PwlVBI=
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEogIBAAKCAQEA1UihMG7GW/QrsYX1PqRfu9is9MwgDkQFkZ8EXyLceWdZol1P
      MwObIBggPOojV8MqpuZhDTUmsObAi95Ih4Zinkzx5bQWwfJOjWMyoWlR7qPlPpeo
      5aozS4igrR8Jtc6pzKwJRTQ5dCjYw6MASQPRI0sbSLcNJctaZtCMQZ0uK2zL3WJB
      pIPtaQUMgt/C2Jy2Yail4sZlLpyMbQMP3MDk0x+0Pol7IOrswyUkqKmInqMLySZI
      tEVzLuOcVHya5k61k6nPVie4E4ll/fmpb6uu01xgaxMug3m79J7cLEJ5W/dpd0gr
      gA+jWkUFk0JA1Y8BN/lvF7SRDXBnRUPJBomlCQIDAQABAoIBAHrQidED2MEWhxqA
      UsoxiNb4XA/Sgxrz3GkQMko+Hsrh0mGGbZaomukOO3IYOSXOQn885eHfG1p+ZZ2k
      8ArqPVt+X0SwQPKJfuY3mu4LYZyTT+yNFDq7mTIDiNJnrwW77VboMaZh/jt/UeLF
      t+JTilhSZXGnxrBeur29lSYrRmzF+i7X94MakM/LhPtttc9+SwVnw8EWWXsQ6KPg
      3X8c1eiSHemlogOxdTaGp+RtekUu9CEQmTUFRgVUGMcPRYW2fQlhe5WRy4NDZJQD
      l7P/k1LZbxD1131e7QnlK5aOL0ehjt8eDSYFUsnM3BRRDoZ2jZfgcoW0+TaOJ00h
      soHE7iECgYEA7gxUdvKYqFeM3tFtjrhybmPAVqXSec4Vs8vMqHod11GfqZeYjRCh
      RFcsfqped9vwdrquTTU5YAx3m1FqII/ldmOS8r96y+4WKBCqgZ+PwMj2yHXvoeK5
      vf60Z+lIYEWyIiFlAorVuDVP5k6MYNIVe9xLdGqCF5RZgEJ5E+/MmcUCgYEA5V40
      zwd8q5tdT2NA5FQgfun8RcoefVLcpbhcjyMuRdBaS1ed3C/Ea+z0rxbvglPKcCdO
      Kpc4gBa55IbLyecUERLgI3TltR++68SUSE8oUkJC9P+6jKyJfV1nvMfCzJGvnXuZ
      FMT25BTv31hgDn+GrkYic33dUgJW/rRCCJlFxnUCgYAz2cWwHwl3aZKKmi117+El
      AF9NwFLHK0d44URT0j2bv/C6HQYie+PUofRUJsHq86I+Ck2bDaPQswZ+c05dh1pg
      3jGM5H5X2yVCVtb84weJ43w1j9LBfUQeCVO8aojUR3oUixeYZStGM36S2Lsk/72k
      PAjHc8egYTTZhiEfYZU8TQKBgFkmqb4Cefm1lKojuPhGnq7VwaFas0kg6PRc2VUl
      ZnnqRv5b2tbXljAg1OYtMwDkEp5PGLnDN0nKC40Au7sJXypXvNjrOdGRukKnkL8e
      TVzTtnTF5TuCYJpisMyWlGexBKJghRqmiBQU8ztLfWKFPNMGHaPoOxsCH9A2Pihb
      ctx1AoGAGN4WilL6wxw6hpHEsK7NfMDnr8I1x4ieVXcRyhukZmWgRA0HsfeBPeOm
      to4kKPPtJQ6zNKVAWxJTF3XKX3Z4oAnsyi2cUTw2+KBcMApnIZF/vNkkPy/nxgap
      uz1sboIKT01Tk0nSNKL+i6MzTGbfDnCqhdK9RYYaq7yyxI38uBI=
      -----END RSA PRIVATE KEY-----
proxyAgent:
  install: true
  proxyServerIP: 172.xxxxx
  cert:
    clusterCaCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDjTCCAnWgAwIBAgIJANIhggLSSlYtMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDdaGA8yMTIyMDkxODA5MzgwN1owXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgM
      AXgxCjAIBgNVBAcMAXgxCjAIBgNVBAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMM
      AmNhMRAwDgYJKoZIhvcNAQkBFgF4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
      CgKCAQEA4lN3G30VKwtnpqLZiB/0NK+pNtPdPX+/iz21D7EuF8kgpryJ9PecFW7I
      OVMH82GpWqpjPbbVSCVph67lq0kKZ4nW1YsGwkmY9qBZCwGpl/9tj7Txx7xJPaGF
      +uILrfqlDPMEdUBdc5ceZBlsz8xdHWdfFTdXCEwtKQlaSAzUbPINHUQox/A2pGdj
      8p6omiEWlNUPVruexNzdHWJjtKIhh9L0iL0hBb6eNIO/rSqoZ64X9oqjRuQMwX9j
      UXoV3fOQ5yFo/z501DPBRMzI2KGKWT7TqzZqmVNiNpzvvQFeDJ2zFutUHKye3IPN
      0gpktPZ7W6ROW2bYOptLUN0GvP/V2wIDAQABo1AwTjAdBgNVHQ4EFgQUB/FrMnWn
      7Szrwz4coYkGQVH/ei0wHwYDVR0jBBgwFoAUB/FrMnWn7Szrwz4coYkGQVH/ei0w
      DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAvl2lJzozUfFlj2Plowvd
      LB64RXDR78kW0SKfBLSKbpLcgR3O9AkZ17Wgtbcr3EmV7Wmg0vl1JigxgtVVY6ns
      jjeuod13PKOC1O8+a2qfWtixv4Y0y2plAUSYvobDXOcqzfLEs/WudwNi7cDtBRth
      K81O2Wd1k4R7txLup+uESZfwWf2vXJoO8PvTQuyG1ihtdA4P/ZIr7P7u2qAbJzDX
      Mgy79ANhuiPt47g5UNbp25dVGVMFPYViUj1r5zeZQrZK+bC4lzQpQFdY5OXO7DbV
      /b/6uZyrFmLasLyppz0L0I6L0hFsDnbCmjY49GhQePlTz+mLaafbeW9HJiZseePB
      Cw==
      -----END CERTIFICATE-----
    clusterCrt: |
      -----BEGIN CERTIFICATE-----
      MIIDODCCAiCgAwIBAgIJALfJxEK+C6bOMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
      BAYTAnh4MQowCAYDVQQIDAF4MQowCAYDVQQHDAF4MQowCAYDVQQKDAF4MQowCAYD
      VQQLDAF4MQswCQYDVQQDDAJjYTEQMA4GCSqGSIb3DQEJARYBeDAgFw0yMjEwMTIw
      OTM4MDhaGA8yMTIyMDkxODA5MzgwOFowJzEOMAwGA1UEAwwFcHJveHkxFTATBgNV
      BAoMDHN5c3RlbTpub2RlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
      AMTXS3oG3QJEwNCZCwTyDFivlQD6W0lNEPSILk7d3gQLW+Fzgzgnfq7vLTBa4WeI
      8FdNRWG6JjVDrqBLC7kcQ2W//Me8n0qtPyKm8uIwvGtFZX+/0z5mtGrUzWgv6Wz8
      yWI/2tBcGkBj6aUncfWH/yXDzu24IW7mguhXsiUCv1TYqTB9S0SJPVAO6U4VlAtD
      2BY0WK0BoaL2GDPfeBlGb35zExwvTAqFv79WbyHZmpZ0aZ7LavK/EqlUCi/IuWMg
      blpp02kE1vHXwAhak0dBe5i4IoJVlEa5MFz8nxg5o8DL8sl6D+ZRB+BczUOnmVkY
      +r0YApfYBO9R2V+J2CJX6kMCAwEAAaMwMC4wLAYDVR0RBCUwI4cEfwAAAYIKa3Vi
      ZXJuZXRlc4IJbG9jYWxob3N0hwSsEQc5MA0GCSqGSIb3DQEBCwUAA4IBAQBIrH9N
      XYgh1yvd0D2YBh79wQ2GfTQlsegaHxpua8LADZZ0IwFNUN2wSaLZzg3i+pC4kubU
      ILws65pW48U0vHF5EQ6tZ/WGBM6u5k6Q9vkPRX9yT+OPDse5nGJMVLOmlwP65V6o
      mWFako5nvizVRDV5RYiY+KlgtMZpIUUe57yodPDysBcXBXMhQ4Sk4zwBwy5+q2uO
      Ux/Y4O1StxQ5QzTLAgr1vI+VC+yf3ho8Clf5cDbKy6TNwqRLiI0Q7DVU906YZgl/
      kI7UhhvZTCVFwteAVTZFWiwWJKsImypodP1zNc8413hI65pygWjHBYDXX9YR8md9
      FkzRNd1l0T21eP+s
      -----END CERTIFICATE-----
    clusterKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpAIBAAKCAQEAxNdLegbdAkTA0JkLBPIMWK+VAPpbSU0Q9IguTt3eBAtb4XOD
      OCd+ru8tMFrhZ4jwV01FYbomNUOuoEsLuRxDZb/8x7yfSq0/Iqby4jC8a0Vlf7/T
      Pma0atTNaC/pbPzJYj/a0FwaQGPppSdx9Yf/JcPO7bghbuaC6FeyJQK/VNipMH1L
      RIk9UA7pThWUC0PYFjRYrQGhovYYM994GUZvfnMTHC9MCoW/v1ZvIdmalnRpnstq
      8r8SqVQKL8i5YyBuWmnTaQTW8dfACFqTR0F7mLgiglWURrkwXPyfGDmjwMvyyXoP
      5lEH4FzNQ6eZWRj6vRgCl9gE71HZX4nYIlfqQwIDAQABAoIBAQDDahFuNX4XkHXz
      INyFrl5/dOByNhmGLdIVUVTXyv9r+HLg4TJ1/w+PT7cG+FIp3RLtOLMWkBz6aBTm
      4VnkDaEDVJP5HtLs6g1xs1gRId0Mibp4FtH6cy/NlzHMqzQLlPHAbjAnBiixn/8G
      jKBf5wFwVnIKIfBfDs0LKHSctZuI+EkrJ2AQQ5lCDQ3J8GARW5bD1GJ9IZh9m1W8
      eCcIZqKCQhUu9LdNJZKLj1PEfkkEerQv/VEveZVK+udVBavxqeK8aObQuurNwdyG
      xTdW59tzLZ8760u5z6CSjSHlXi8hSPWIdzWKF3In78Yt50soRYZ4svXD2q3aqBlW
      B1U6QF2RAoGBAObku8cVM7YJzVwWPc52PoPnyfpxtbhGWbo8UASDrSAYd1uHEZFW
      tHOd1Jropq5oVDSShaV7r9RABR0cRp5UF4asQCIgwxKqE+1I6O1i0YlN8HTP398W
      +K+rzpsLEGI9bigoQlUqzron4sE2iZOHBmFfA2OMQSqIsY5NRAL7M3FFAoGBANo+
      qLkPmRrf5bdW1wAluZnX+y4KjZNSGROH7uDFNcw8O9LWivi2xeVUB4BmLCQc7KLk
      931YvFQe8QjrIljJ/z9mSPf3gILsE2PJXEQxa0lO3Ck1nAJ1uLPdwNd2GXnMHGS5
      GzfwCJITqPLGpA7vYi/8vPRCji6jQpudRoky7LHnAoGARc9sswHHE8gc7jXPVChO
      OIWRfQZ6Yg/rOzLXIrZJVwXuuJ67Ag2MG9ThnkCvfNYQd6DBhcpC3FLK+QU+mMpm
      HpLgBTGQqYkA42KDAps5LWGCTV84mfQRFtREcH5E4t8d+khJ3gRGYuSCW9bIe/F8
      TV5z4NpcLacMwDC+WoRnBl0CgYEAvJ3qtAYHDHL01b73kekIvy0qqF/2e5w4j+ir
      Sxclp51nFpDVCp7r6Yb19MJY490bpqy8OFi4avtS1aJhzNrG/sQnzCbbqkrddyBP
      3Z/YWUMOg48Rhg630ruULqKi4eDFqUdRfsa0oYrcOPX0U/QPp+mGZJQgza3LozIK
      BAk/B4ECgYAhLMj7zy9pTlsvT38lKdSyDVGCg2WQIDJV1owL78eJ3LruPoGSYl/I
      jYnDEhGCzg9AlqMnL8PhWD+6zwrWVvxBR3LT5qy5v26BR+KNum9AU7Pi/2NlyicU
      JQtevMj8wf7/rYvnE9ryiiyH0jEm5SRKFqHTkMlWcMQit5ggQHlJnw==
      -----END RSA PRIVATE KEY-----

host

rm -rf /var/lib/karama-system

helm install karmada  -n karmada-system --create-namespace  karmada-1.3.0.tgz -f host.yaml

agent

helm install karmada-agent -n karmada-system --create-namespace  karmada-1.3.0.tgz -f agent.yaml

我可以邀请 @Poor12 来看一下，但我不确定这些信息足够能够复现，因为具体修改内容还是看不到的。

我用的helm chart 包已经上传了，你对比1.3的就知道我改了啥

不过1.3的chart确实有问题，刚开始连部署都失败。。。，之后我会提PR

奇怪，目前chart安装在CI中是有覆盖的。欢迎提交issue，把报错信息贴出来。

主要是自定义证书要传ca.key, post-install-job 有些yaml错位

这个是怎么自定义的，是在chart安装时使用配置参数？

主要是这两个地方。。。

kube-controller-manager报错找不到key no such file 之类的

导致post-install-job执行失败根本安装不了

到时候我提一下PR，贡献一下

@RainbowMango 用了证书auto的一样问题，报503

@liangyuanpeng: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/karmada-io/karmada/issues/2628#issuecomment-1278367435): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

@631068264: Reopened this issue.

In response to [this](https://github.com/karmada-io/karmada/issues/2628#issuecomment-1278367790): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

我刚刚在我的环境中也重现了这个问题。

我的控制面主机ip是10.10.103.67，子集群的主机ip是10.10.103.68

用karmadactl init初始化的集群，当我设置karmada-agent的参数--cluster-api-endpoint=https://127.0.0.1:6443时，就访问不了了，主要这个127.0.0.1在容器中确实访问不了karmada-agent，所以报错了。

- command:
        - /bin/karmada-agent
        - --karmada-kubeconfig=/etc/kubeconfig/karmada-kubeconfig
        - --cluster-name=member68
        - --cluster-api-endpoint=https://127.0.0.1:6443
        - --proxy-server-address=http://10.10.103.67:8088
        - --cluster-provider=
        - --cluster-region=
        - --enable-cert-rotation=true
        - --cluster-status-update-frequency=10s
        - --bind-address=0.0.0.0
        - --secure-port=10357

[root@master67 ~]# karmadactl get pod --kubeconfig /etc/karmada/karmada-apiserver.config  -v=4
error: cluster(member68) is inaccessible, please check authorization or network

如果设置--cluster-api-endpoint=https://10.96.0.1:443:6443，其中https://10.96.0.1:443是子集群的kube-apiserver的service ip

 - command:
        - /bin/karmada-agent
        - --karmada-kubeconfig=/etc/kubeconfig/karmada-kubeconfig
        - --cluster-name=member68
        - --cluster-api-endpoint=https://10.96.0.1:443
        - --proxy-server-address=http://10.10.103.67:8088
        - --cluster-provider=
        - --cluster-region=
        - --enable-cert-rotation=true
        - --cluster-status-update-frequency=10s
        - --bind-address=0.0.0.0
        - --secure-port=10357
        - --v=4

设置proxy-agent的'--agent-identifiers=host=10.244.0.134'，其中10.244.0.134是karmada-agent pod的ip

 args:
            - '--ca-cert=/var/certs/agent/ca.crt'
            - '--agent-cert=/var/certs/agent/proxy-agent.crt'
            - '--agent-key=/var/certs/agent/proxy-agent.key'
            - '--proxy-server-host=10.10.103.67'
            - '--proxy-server-port=8091'
            - '--agent-identifiers=host=10.244.0.134'

这样也可以模拟出单方面通信的情况，这样就可以使用代理了。

[root@master67 ~]# karmadactl get pod --kubeconfig /etc/karmada/karmada-apiserver.config  -v=4
NAME                     CLUSTER    READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-7x6mh   member68   1/1     Running   0          27h

proxy-server的日志

I1014 14:40:10.139945       1 server.go:616] "Connect request from agent" agentID="dcf15a6c-169a-4a1b-9ed6-cc9b5a0dcbed"
I1014 14:40:10.140154       1 backend_manager.go:191] "Register backend for agent" connection=&{ServerStream:0xc000292180} agentID="10.244.0.134"
E1014 14:40:11.298876       1 server.go:650] "stream read failure" err="rpc error: code = Canceled desc = context canceled"
I1014 14:40:11.298996       1 backend_manager.go:221] "Remove connection for agent" connection=&{ServerStream:0xc0001f00c0} identifier="10.10.103.68"
I1014 14:40:11.299091       1 server.go:669] "Close frontends connected to agent" count=0 agentID="40bb69a7-fd5a-4d4f-9454-60ae1ebca107"

@lonelyCZ

设置proxy-agent的'--agent-identifiers=host=10.244.0.134'，其中10.244.0.134是karmada-agent pod的ip

....这不科学啊，pod IP很容易变得啊

--cluster-api-endpoint
和
--agent-identifiers

那为什么以前1.2版本直接用主机IP就可以？

@lonelyCZ 不行，而且我也不太确定你们组件镜像版本号。。。。因为都是lastest 好迷幻。。。。我更新下镜像试试

agent

        - /bin/karmada-agent
        - --karmada-kubeconfig=/etc/kubeconfig/kubeconfig
        - --cluster-name=member1
        - --cluster-api-endpoint=https://10.43.0.1:443
        - --proxy-server-address=http://172.xxxx:8088
        - --cluster-status-update-frequency=10s
        - --leader-elect-resource-namespace=karmada-system
        - --bind-address=0.0.0.0
        - --secure-port=10357

proxy-agent

        - --ca-cert=/var/certs/agent/ca.crt
        - --agent-cert=/var/certs/agent/proxy-agent.crt
        - --agent-key=/var/certs/agent/proxy-agent.key
        - --proxy-server-host=172.xxxx
        - --proxy-server-port=8091
        - --agent-identifiers=host=10.42.0.60

I1015 16:52:24.045692    9071 loader.go:372] Config loaded from file:  karmada-apiserver.config
I1015 16:52:34.053477    9071 round_trippers.go:553] GET https://172.xxxxxx:5443/apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api 503 Service Unavailable in 10007 milliseconds
error: cluster(member1) is inaccessible, please check authorization or network

更新了下镜像。。。。好了:)，用的是原来我用的部署方式

 karmadactl get pod --kubeconfig karmada-apiserver.config 
NAME                     CLUSTER   READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-txqzt   member1   1/1     Running   0          28s
nginx-6799fc88d8-b6nmd   member2   1/1     Running   0          28s
nginx-6799fc88d8-zlvg6   member2   1/1     Running   0          28s

@lonelyCZ

设置proxy-agent的'--agent-identifiers=host=10.244.0.134'，其中10.244.0.134是karmada-agent pod的ip

....这不科学啊，pod IP很容易变得啊
--cluster-api-endpoint
和
--agent-identifiers
那为什么以前1.2版本直接用主机IP就可以？

这只是我用来模拟控制面无法直接访问karmada-agent的例子，实际使用还是推荐使用主机ip的

@lonelyCZ 请问一下你们是修复了什么地方导致503好了？之后出现类似的问题怎样排查？

请问一下你们是修复了什么地方导致503好了？之后出现类似的问题怎样排查？

你之前用的镜像是哪个版本的？

刚开始发现问题时候是1.2 发现用自定义证书时候 agent 注册失败说 clusters.cluster.karmada.io not found之类的

然后10月12号尝试用1.3的试试，拉了全部最新镜像，发现可以agent注册了,但是会503报错

然后今天再拉一次镜像，重新安装一下好了。。。

然后10月12号尝试用1.3的试试，拉了全部最新镜像，发现可以agent注册了,但是会503报错然后今天再拉一次镜像，重新安装一下好了。。。

我用的是1.3的镜像是可以的，你要不换成v1.3版本的镜像再试一下？

现在是可以了，但是12号的 1.3的镜像不行，主要是不知道今天的和12号的镜像有什么差别，而且helm chart 包里面的都是

swr.ap-southeast-1.myhuaweicloud.com/karmada/xxx:latest

这样的建议改成

swr.ap-southeast-1.myhuaweicloud.com/karmada/xxx:v1.3

之类的

镜像一般发布后不会使用同版本tag对已有镜像进行覆盖, 你可以用 docker inspect {imageName} 检查下 Created 字段.

而且helm chart 包里面的都是 swr.ap-southeast-1.myhuaweicloud.com/karmada/xxx:latest

是的，helm chart里面镜像使用latest绝对不是最佳实践，很容易对已经部署的应用产生未知的影响，例如假设 v2.x 的版本 karmara 删除了某个配置项，而已部署的 1.x karmada应用在启动一个新的 pod 时拉取了 latest 镜像，会由于没有这个配置项而启动失败，我认为这是一个很不友好的问题。

在 Helm 官网翻看了下最佳实践，可以看到 Helm官网已经有对应声明。

A container image should use a fixed tag or the SHA of the image. It should not use the tags latest, head, canary, or other tags that are designed to be "floating".

Images may be defined in the values.yaml file to make it easy to swap out images.

https://helm.sh/docs/chart_best_practices/pods/#images

/cc @Poor12 @RainbowMango

我从 https://raw.githubusercontent.com/karmada-io/karmada/master/charts/index.yaml 看到karmada已经有发布的 helm chart，随意查看了下v1.2.1和v1.2.2发现使用的镜像是latest, 如果旧版本的 crd 与最新的latest 版本容器无法一起工作，那么这些已经发布的 helm chart 不仅是无效的(默认情况下)还会对用户造成困惑.

不确认目前的做法是否有特别的原因?

This is indeed a problem. We hope that the mirror of the master branch will always be the latest. For those released releases, we need to manually change the mirror tag to the tag of the current version.

This is indeed a problem. We hope that the mirror of the master branch will always be the latest. For those released releases, we need to manually change the mirror tag to the tag of the current version.

Could we plan to release chart, like to https://artifacthub.io/, a public hub?

karmada-io / karmada

ANP cluster is inaccessible, please check authorization or network /proxy/api 503 Service Unavailable #2628