k8s 리소스 순서대로 배포하기

[BaeKY]

https://github.com/kubernetes/kubectl/issues/1117 같은 문제를 겪는 사람...

kfilt

• 위 이슈에서는 kfilt를 사용하였지만 yq로도 충분히 가능할것 같다

yq

• 아래와 같이 하면 동작할것 같은데...

  # CRD만 먼저 생성
  cat my-manifest.k8s.yaml | yq '. | select(.kind == "CustomResourceDefinition")' | kubectl apply -f -
  
  # 경건한 마음으로 CRD 생성을 기다리자
  sleep 27
  
  # CRD 제외한 나머지 생성
  cat my-manifest.k8s.yaml | yq '. | select(.kind != "CustomResourceDefinition")' | kubectl apply -f -

• 복잡하게 kubectl wait --for condition=~~~ 해서 하는것 보다 간단할듯.
• sleep time 정도만 parameter로 받도록 하자.(기본값 27)

[BaeKY]

script 작성

• deploy-manifest.sh

  #!/bin/bash
  
  MANIFESTS=$1
  
  case $MANIFESTS in
  -*|--*|'')
    echo "Unknown k8s manifest path '$MANIFESTS'"
    exit 1
    ;;
  esac
  shift
  
  if [ -d $MANIFESTS ]; then
  MANIFESTS=$(realpath $MANIFESTS)/*
  elif [ ! -f $MANIFESTS ]; then
  echo "Files not exists"
  exit 1
  fi
  
  POSITIONAL_ARGS=()
  
  while [[ $# -gt 0 ]]; do
  case $1 in
    -w|--wait-crd-creation)
      WAIT_CRD_CREATION="$2"
      shift # past argument
      shift # past value
      ;;
    -*|--*)
      echo "Unknown option $1"
      exit 1
      ;;
    *)
      POSITIONAL_ARGS+=("$1") # save positional arg
      shift # past argument
      ;;
  esac
  done
  
  set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters
  
  cat $(ls $MANIFESTS | grep '.yaml') | yq '. | select(.kind == "CustomResourceDefinition")' | kubectl apply -f -
  
  sleep $WAIT_CRD_CREATION
  
  cat $(ls $MANIFESTS | grep '.yaml') | yq '. | select(.kind != "CustomResourceDefinition")' | kubectl apply -f -

Metallb 배포 테스트

• 아래 명령어로 deploy-manifest.sh를 통해 ./manifest/0000-metallb.k8s.yaml를 배포한다

  # cd <root>/chart
  bash ./scripts/deploy-manifest.sh ./manifest/0000-metallb.k8s.yaml --wait-crd-creation 30

• 결과: 실패

  customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
  serviceaccount/l2-lb-metallb-controller created
  serviceaccount/l2-lb-metallb-speaker created
  secret/webhook-server-cert created
  clusterrole.rbac.authorization.k8s.io/l2-lb-metallb:controller created
  clusterrole.rbac.authorization.k8s.io/l2-lb-metallb:speaker created
  clusterrolebinding.rbac.authorization.k8s.io/l2-lb-metallb:controller created
  clusterrolebinding.rbac.authorization.k8s.io/l2-lb-metallb:speaker created
  role.rbac.authorization.k8s.io/l2-lb-metallb-pod-lister created
  role.rbac.authorization.k8s.io/l2-lb-metallb-controller created
  rolebinding.rbac.authorization.k8s.io/l2-lb-metallb-pod-lister created
  rolebinding.rbac.authorization.k8s.io/l2-lb-metallb-controller created
  service/metallb-webhook-service created
  daemonset.apps/l2-lb-metallb-speaker created
  deployment.apps/l2-lb-metallb-controller created
  validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
  Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": dial tcp 10.103.248.228:443: connect: connection refused

• CRD생성이 문제가 아니었다
  • https://github.com/metallb/metallb/issues/1540#issuecomment-1199180633
  • 위 이슈를 잘 살펴보니 metallb-webhook-configuration이 먼저 생성되어서 webhook-server-cert secret이 먼저 만들어져야하는듯
  • 그리고 에러로그도 자세히 봤더니...

    Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": ~~~~~~~~~~

에러로그를 잘 확인하자...

[BaeKY]

개선

• CRD를 먼저 생성하고 기다리는것이 아니라 사용자가 "나중에 생성할 Resource들을 지정"하는 방식으로 배포하도록 한다

[BaeKY]

결과

• metallb 다시 배포

  pnpm k8s:deploy ./manifest/0000-metallb.k8s.yaml --wait 17 --resources IPAddressPool

• 결과: 성공

  > @kube-ops/chart@1.0.0 k8s:deploy /Users/bky/dev-private/k8s-lab/chart
  > ./scripts/deploy-manifest.sh $@ "./manifest/0000-metallb.k8s.yaml" "--wait" "17" "--resources" "IPAddressPool"
  
  IPAddressPool
  serviceaccount/l2-lb-metallb-controller created
  serviceaccount/l2-lb-metallb-speaker created
  secret/webhook-server-cert created
  customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
  customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
  clusterrole.rbac.authorization.k8s.io/l2-lb-metallb:controller created
  clusterrole.rbac.authorization.k8s.io/l2-lb-metallb:speaker created
  clusterrolebinding.rbac.authorization.k8s.io/l2-lb-metallb:controller created
  clusterrolebinding.rbac.authorization.k8s.io/l2-lb-metallb:speaker created
  role.rbac.authorization.k8s.io/l2-lb-metallb-pod-lister created
  role.rbac.authorization.k8s.io/l2-lb-metallb-controller created
  rolebinding.rbac.authorization.k8s.io/l2-lb-metallb-pod-lister created
  rolebinding.rbac.authorization.k8s.io/l2-lb-metallb-controller created
  service/metallb-webhook-service created
  daemonset.apps/l2-lb-metallb-speaker created
  deployment.apps/l2-lb-metallb-controller created
  validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
  ipaddresspool.metallb.io/local-ip created

[dev-whoan]

감사합니다. metallb 최신 버전 에러 발생 이유가 배포 순서 때문이었군요....