Request to have kustomize agent image

[xiabai84]

For deployment of multiple Microservices could be a template engine like kustomize very useful.

The main scope of using it are:

1. It can collect multiple Kubernetes yaml templates and build them together into one single deployment script
2. it can also patch the changes of deployments, if your application is required to be deployed in multiple environments.

Current problem: If an application is build from multiple Microservices, with the current version of Kubesphere we must create for each service a separate Jenkins job. With the help of kustomize you can create a deployment yaml template, which contains all deployments for this application. Then can Jenkins take exactly this single template and deploy all services with only one job.

For multiple deployment environments you can create a overlay patch for special configurations, the deployment file structure would be like:

├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
└── overlays
    ├── dev
    │   ├── kustomization.yaml
    │   └── dev_patch_deployment.yaml
    ├── test
    │   ├── kustomization.yaml
    │   └── test_patch_deployment.yaml
    └── prod
        ├── kustomization.yaml
        └── prod_patch_deployment.yaml

See links below: https://github.com/kubernetes-sigs/kustomize https://kustomize.io/

[xiabai84]

Dockerfile would be like this？

FROM kubespheredev/builder-base:v3.1.0

ENV KUSTOMIZE_VER 4.3.0

RUN cd && \
    curl -sLO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv${KUSTOMIZE_VER}/kustomize_v${KUSTOMIZE_VER}_linux_amd64.tar.gz && \
    tar xzf kustomize_v*.tar.gz && \
    mv kustomize /opt/kustomize && \
    ln -s /opt/kustomize /usr/local/bin/kustomize && \
    rm kustomize_v*.tar.gz

CMD ["/usr/local/bin/kustomize"]

[LinuxSuRen]

Great! You are almost done with this. Please consider creating a PR against this.

[JohnNiang]

https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ https://github.com/kubesphere/devops-agent/pull/42#discussion_r715952148

Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command:

kubectl kustomize <kustomization_directory>

To apply those Resources, run kubectl apply with --kustomize or -k flag:

kubectl apply -k <kustomization_directory>

[xiabai84]

Hi, Sorry, I have to reopen this issue, because the current kubectl's built-in kustomize command only supports patching and applying kustomize files. If I want to use for examplekustomize edit set image, kubectl kustomize can't help me further...

In my use case, I'd like to use kustomize in the jenkins builder-container to patch the kustomization.yaml with special git-commit hash for deployment (this kustomization.yaml file is also checked in git repo and contains git-commit-hashes of all my microservices image versions).

Following part are not included in the kubectl kustomize, except "build".

Manages declarative configuration of Kubernetes.
See https://sigs.k8s.io/kustomize

Usage:
  kustomize [command]

Available Commands:
  build                     Build a kustomization target from a directory or URL.
  cfg                       Commands for reading and writing configuration.
  completion                Generate shell completion script
  create                    Create a new kustomization in the current directory
  edit                      Edits a kustomization file
  fn                        Commands for running functions against configuration.
  help                      Help about any command
  version                   Prints the kustomize version

Flags:
  -h, --help          help for kustomize
      --stack-trace   print a stack-trace on error

Additional help topics:
  kustomize docs-fn                   [Alpha] Documentation for developing and invoking Configuration Functions.
  kustomize docs-fn-spec              [Alpha] Documentation for Configuration Functions Specification.
  kustomize docs-io-annotations       [Alpha] Documentation for annotations used by io.
  kustomize docs-merge                [Alpha] Documentation for merging Resources (2-way merge).
  kustomize docs-merge3               [Alpha] Documentation for merging Resources (3-way merge).
  kustomize tutorials-command-basics  [Alpha] Tutorials for using basic config commands.
  kustomize tutorials-function-basics [Alpha] Tutorials for using functions

[JohnNiang]

Hi @xiabai84 , thanks for your feedback! I confirmed that the kubectl kustomize cannot support edit and most built-in commands in kustomize.

So, it's reasonable to add kustomize into builder-base. I'm going to reopen the PR

• https://github.com/kubesphere/devops-agent/pull/42

Besides, if you are interested in no builders, please refer to

• https://github.com/kubesphere/devops-agent/issues/26

for more.

References:

• https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
• https://github.com/kubernetes-sigs/kustomize#kubectl-integration

[xiabai84]

Hi @JohnNiang,

I agree with you, that

1. Kubesphere as platform provider shouldn't provide all kinds of build containers for every programing language and different versions of them. But a platform should give users a secure way to customize build containers based on the base-container, which must be provided from kubesphere. And this customized build container should be stored and versioned in a cluster wide container registry like harbor.

2. in my opinion cluster admin or jenkins admin must take on charge for building such customized build-containers. Cause of potential security issues, I wouldn't delegate such tasks to development team. The current way to customize container is in my opinion too dangerous, because you can basically inject everything from internet in your run time container.

Anyway I think kustomize should be provided as a part of base container. Should I make a PR？

[LinuxSuRen]

+1 from to put the command customize into the base image.