GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
15k stars 1.62k forks source link

Use operator kubectl manifests and the manifests that depend on them. #7152

Open unj1m opened 2 years ago

unj1m commented 2 years ago

Expected behavior

Execute two (or more, obviously) manifests:

The operator and the requested resources are defined.

Example:

skaffold.yaml:

apiVersion: skaffold/v2beta26
kind: Config
metadata:
  name: compute-service
deploy:
  kubectl:
    manifests:
      - https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
      - zclickhouse.yaml

zclickhouse.yaml;

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "dev"
spec:
  configuration:
    clusters:
      - name: "simple"

Equivalent kubectl command (more or less):

kubectl apply \
  -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml \
  -f zclickhouse.yaml

I've tried disabling validation in both scaffold and kubectl, to no avail.

Actual behavior

Scaffold spews the contents of the first manifest for some reason and then outputs:

 - stderr: "error: unable to recognize \"/home/jim/p/compute-service/clickhouse.yaml\": no matches for kind \"ClickHouseInstallation\" in version \"clickhouse.altinity.com/v1\"\n"
 - cause: exit status 1

Running kubectl directly, gives the same error:

error: unable to recognize "clickhouse.yaml": no matches for kind "ClickHouseInstallation" in version "clickhouse.altinity.com/v1"

If I only apply the first manifest with skaffold (or kubectl) and then apply the second manifest manually, it works.

Information

apiVersion: skaffold/v2beta26
kind: Config
metadata:
  name: compute-service
deploy:
  kubectl:
    manifests:
      - https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
      - clickhouse.yaml

Steps to reproduce the behavior

skaffold dev -vdebug

Output:

DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v1.35.2 ConfigVersion:skaffold/v2beta26 GitVersion: GitCommit:f8ae4e65bafdcfd39e4de67329b185432899c7ad BuildDate:2022-01-13T16:13:49Z GoVersion:go1.17 Compiler:gc Platform:linux/amd64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/home/jim/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /home/jim/p/compute-service/skaffold.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: kind-kind              subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v1.35.2  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0000] push value not present in NewBuilder, defaulting to false because cluster.PushImages is false  subtask=-1 task=DevLoop
INFO[0000] build concurrency first set to 0 parsed from *local.Builder[0]  subtask=-1 task=DevLoop
INFO[0000] final build concurrency value is 0            subtask=-1 task=DevLoop
Listing files to watch...
INFO[0000] List generated in 22.244µs                    subtask=-1 task=DevLoop
Generating tags...
INFO[0000] Tags generated in 4.197µs                     subtask=-1 task=Build
Checking cache...
INFO[0000] Cache check completed in 8.771µs              subtask=-1 task=Build
Tags used in deployment:
Starting deploy...
DEBU[0000] getting client config for kubeContext: `kind-kind`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl version --client -ojson]  subtask=0 task=Deploy
DEBU[0000] Command output: [{
  "clientVersion": {
    "major": "1",
    "minor": "23",
    "gitVersion": "v1.23.4",
    "gitCommit": "e6c093d87ea4cbb530a7b2ae91e54c0842d8308a",
    "gitTreeState": "clean",
    "buildDate": "2022-02-18T02:26:27Z",
    "goVersion": "go1.17.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
]  subtask=0 task=Deploy
DEBU[0000] Running command: [kubectl --context kind-kind create --dry-run=client -oyaml -f /home/jim/p/compute-service/zclickhouse.yaml -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml]  subtask=0 task=Deploy
Cleaning up...
DEBU[0000] Running command: [kubectl --context kind-kind create --dry-run=client -oyaml -f /home/jim/p/compute-service/zclickhouse.yaml -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml]  subtask=-1 task=DevLoop
WARN[0000] deployer cleanup:kubectl create: running [kubectl --context kind-kind create --dry-run=client -oyaml -f /home/jim/p/compute-service/zclickhouse.yaml -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml]
 - stdout: "... contents of first manifest ellided ..."
  - stderr: "error: unable to recognize \"/home/jim/p/compute-service/zclickhouse.yaml\": no matches for kind \"ClickHouseInstallation\" in version \"clickhouse.altinity.com/v1\"\n"
 - cause: exit status 1
DEBU[0010] exporting metrics                             subtask=-1 task=DevLoop
tejal29 commented 2 years ago

thanks @unj1m. This is an known issue where skaffold can't relies on kubectl.
There are couple of ways to solve this 1) Define modules and include your the operator definition in the module A. Module B will contain the resource and will depend on module B.

The other way is actually use kpt deployer. Kpt has the ability to detect dependencies and then apply the operator first and resource later.

Happy to know your thots.