Open MurzNN opened 11 months ago
I think we should just remove the samples inventories (or at least the variables) and rely on roles defaults (also for documentation).
For some data point, when I came to work on the clusters I manage with kubespray, I removed more than 10k lines of inventory, accross 8 clusters. Most of it was just default value, which made quite hard to tell what was not default.
I finally found a way to minimize the amount of files, to store only values changed from defaults, here is all that we needed:
- hosts.yaml
- group_vars/all.yaml
And the example content - hosts.yaml
:
all:
vars:
ansible_connection: ssh
ansible_user: root
hosts:
node1:
# The hostname to connect, or an external IP address of the node.
ansible_host: mycluster.mydomain.com
# Optionally, you can override the IP address for connecting to the node, if it differs from the hostname.
# access_ip: 1.2.3.4
# The node IP address in the internal network. Uncomment and set if you have a separate internal network.
# ip: 10.0.0.2
children:
kube_control_plane:
hosts:
node1:
kube_node:
hosts:
node1:
etcd:
hosts:
node1:
k8s_cluster:
children:
kube_control_plane:
kube_node:
and group_vars/all.yaml
:
cluster_name: mycluster.mydomain.com
supplementary_addresses_in_ssl_keys:
- node1.mycluster.mydomain.com
helm_enabled: true
metrics_server_enabled: true
metrics_server_kubelet_insecure_tls: true
krew_enabled: true
kubectl_alias: k
And here is a "best practice" structure of the git repo:
├── inventory
│ └── mycluster
│ ├── group_vars
│ │ └── all.yaml
│ └── hosts.yaml
├── kubespray (as a git submodule)
└── .gitmodules
Having this, we can simply deploy the cluster using a command:
cd kubesprary && ansible-playbook -i ../inventory/mycluster ./cluster.yml
So, if this approach looks okay, maybe add this example as a best practice to manage minimal Kubespray configuration?
Maybe you could add group specific files in that example.
For example supplementary_addresses_in_ssl_keys
should rather go in a group_vars/kube_control_plane.yml
file rather than all (it's only used in the control-plane role)
Maybe you could add group specific files in that example. For example
supplementary_addresses_in_ssl_keys
should rather go in agroup_vars/kube_control_plane.yml
file rather than all (it's only used in the control-plane role)
I can, but then I should distribute all other values to different files too, and this will end up with a bunch of files again, instead of one convenient file with few strings. And having this in the all.yaml
file will give us no conflicts. So I vote to keep this as simple as possible.
You're not wrong, that's enough for "Get started" type of doc. We can always are more advanced examples later on.
I created an example repository with a minimal template of the Kubespray configuration here: https://github.com/Murz-K8s/kubespray-inventory-base Feel free to discuss and improve it.
Also, I have a question - can we simplify this part of the hosts.yaml
file?
children:
# The list of hosts that should be included into the cluster
kube_node:
hosts:
node1:
# node2:
# The list of nodes that should be used as the control plane.
kube_control_plane:
hosts:
node1:
# The list of nodes that should be used as the etcd storage.
etcd:
hosts:
node1:
# What's the purpose of this?
k8s_cluster:
children:
kube_control_plane:
kube_node:
Can we somehow get rid of the all.children.k8s_cluster.children
group?
the k8s_cluster
group is used all over the place in plays, not sure how practical it would be to remove it from inventories... Maybe we could compute it at the same time that we handle "legacy groups" :thinking:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/lifecycle frozen /remove-lifecycle rotten /help /good-first-issue
@VannTen: This request has been marked as suitable for new contributors.
Please ensure that the issue body includes answers to the following questions:
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue
command.
What would you like to be added: Please add an official "best practices" instruction on how to manage the Kubespray configuration for a custom cluster in a separate git repository, with the minimum amount of files.
Why is this needed: Now a lot of instructions and articles about Kubespray recommend copying the whole directory inventory/sample to
inventory/myclustername
and doing the modifications of the needed options directly in these files.Yes, this works well!
But, as a result, most of the users just adds this
inventory/mycluster
directory to the git, and upload all Kubespray folder to the git, producing a whole copy of Kubespray repo with a lot of files.Even if some users are smart and store in a custom git repository files only from the
inventory/mycluster
directory, there are still a lot of files! Look:Usually, users change only a couple of lines in the
group_vars/k8s_cluster/k8s-cluster.yml
from the default values, like:and in the
group_vars/k8s_cluster/addons.yml
:So, they can simply create a single file like
mycluster-defaults-overrides.yml
and just put these couple of changes here, and store only a single file with 6 lines in their custom git repo, and do not upload a hundredth copy of the whole "sample" directory to github.com, wasting the cloud storage space for nothing.Moreover, a benefit of this approach will be a quick finding out what exact parameters were changed from defaults for this cluster, instead of digging into dozens of files to search which exact couple of lines were changed from defaults.
I created an example repository with a minimal template of the Kubespray configuration here: https://github.com/Murz-K8s/kubespray-inventory-base Feel free to discuss and improve it.