Upgrade failed when using patches directory

kaviankarimzadeh commented 3 hours ago

What happened?

We have some args already added to our kube-system StaticPods like etcd, kube-apiserver that needs to be persist during upgrade. until now we were passing --config flag with a path to a file include ClusterConfiguration and all configs that must be persist. But some of these args like encryption-provider-config in apiserver or listen-metrics-urls in etcd gets removed when we run upgrade, so we need to manually add/update those values during upgrade process so the upgrade can continue. And also using --config flags is not recommended for upgrade and soon it'll be deprecated.

so we decided to use --patches flags, but it seems none of the patchStrategy (merge, strategic, json) are capable of adding args to the current args already exist in those StaticPods.

here's an example with merge patchStrategy:

file name: etcd0+merge.yaml / etcd0+strategic.yam

apiVersion: v1
kind: Pod
metadata:
  name: etcd
spec:
  containers:
  - name: etcd
    command:
      - /usr/local/bin/etcd
    args:
      - --quota-backend-bytes=8589934592
      - --listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381

here's an example with json patchStrategy :

 file name: etcd0+json.json

 [
    {
        "op": "add",
        "path": "/spec/containers/0/command/-",
        "value":  [ "--quota-backend-bytes=8589934592" ]
    },
    {
        "op": "add",
        "path": "/spec/containers/0/command/-",
        "value":  [ "--listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381" ]
    }
]

Upgrade command:

kubeadm upgrade apply v1.29.8  --patches /etc/kubernetes/patches/ --dry-run

No args added to to the StaticPod yaml file.

What did you expect to happen?

We're expecting to add additional args to kube-system StaticPods during upgrade process while keeping what is already there in their specs. ( not overriding )

How can we reproduce it (as minimally and precisely as possible)?

here's an example with merge patchStrategy:

file name: /etc/kubernetes/patches/etcd0+merge.yaml

apiVersion: v1
kind: Pod
metadata:
  name: etcd
spec:
  containers:
  - name: etcd
    command:
      - /usr/local/bin/etcd
    args:
      - --quota-backend-bytes=8589934592
      - --listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381

 kubeadm upgrade apply v1.29.8  --patches /etc/kubernetes/patches/ --dry-run

Anything else we need to know?

Here in this document an example provided to check available patchStrategy for PodSpec:

https://v1-29.docs.kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#notes-on-the-strategic-merge-patch

"io.k8s.api.core.v1.PodSpec": {
    ...,
    "containers": {
        "description": "List of containers belonging to the pod.  ...."
    },
    "x-kubernetes-patch-merge-key": "name",
    "x-kubernetes-patch-strategy": "merge"
}

But hew in OpemApi spec for args there's no patchStrategy provided: https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json

    "io.k8s.api.core.v1.Container": {
      "description": "A single application container that you want to run within a pod.",
      "properties": {
        "args": {
          "description": "Arguments to the entrypoint. The container image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell",
          "items": {
            "type": "string"
          },
          "type": "array",
          "x-kubernetes-list-type": "atomic"
        },

Kubernetes version

Current: 1.28.4 Upgrade to : 1.29.8

Cloud provider

OS version

NAME="AlmaLinux"
VERSION="8.10 (Cerulean Leopard)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.10 (Cerulean Leopard)"

uname -a
Linux kubm01 4.18.0-553.el8_10.x86_64 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

containerd://1.6.31

Related plugins (CNI, CSI, ...) and versions (if applicable)

k8s-ci-robot commented 3 hours ago

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

/sig <group-name>
/wg <group-name>
/committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 3 hours ago

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

kaviankarimzadeh commented 2 hours ago

Wrong repo, just submitted same request in kubeadm repository.

kubernetes / kubernetes