k3s production use considerations (and validation)

[andy108369]

@chainzero created the k3s method of provider installation, described here https://akashengineers.xyz/provider-build-scripts

Before getting this to the Production use the following points must be considered, addressed/verified to be supported with the k3s K8s cluster deployment method:

• [x] scale one all-in-one node installation into three-(or more)-node installation
  • [x] consider etcd can be scaled (to avoid SPOF)
  • [x] consider control-plane can be scaled
  • [x] consider safe scaling down as well (in case of node replacement or simply the provider decides to shrink the amount of nodes for whatever reason)
• [x] document node addition (additionally, consider a scenario where that node will be running etcd instance or/and control-plane);
• [x] document node removal (additionally, consider a scenario where that node is running etcd instance or/and control-plane);
• [x] CNI plugins: Is Calico the main & default K8s networking?
• [ ] CNI plugins/calico: Consider installation scenario where one would want to specify K8s internal networking as well, primarily for the performance sake (for internal K8s services/apps communication, including Rook-Ceph persistent storage which can be really heavy on the traffic if it is not done via the internal networking which will lead to significant performance lag and bill if provider's traffic is metered)
• [ ] customize nodefs & imagefs locations: similarly to how it's described here
• [ ] consider etcd backup & restore procedure (kubespray does this automatically each time you run it against your K8s cluster)
• [ ] consider etcd performance - AFAIK, k3s uses sqlite3 DB for the etcd; so there should be some quick perf test for it such as etcdctl check perf we have here

Additioanlly/Ideally

• [ ] custom K8s configs for the nodefs & imagefs thresholds (ref)
• [ ] custom K8s configs for the max. number of container log files that can be present for a container kubelet_logfiles_max_nr, as well as the max.size of the container log file before it is rotated kubelet_logfiles_max_size (ref)

[jigar-arc10]

Here is what we found so far from our testing.

• While scaling down a node, we tried to use the draining method, but akash-services/operator-inventory-hardware-discovery will cause an issue as it is not a DeamonSet. We should look into it. Force draining worked.
• These scripts required root access to the server.
• Successfully tested worker node removal - no issue found.
• Successfully tested control-plane node removal - no issue found.
• Going from single-node to multiple-node provider - no issue found.
• The recommended OS is Ubuntu. It failed for Debian at the ingress-nginx installation stage.
• So far calico is doing great, we will continue monitoring networking issue if any.

We will continue testing further and will report new findings.

[chainzero]

@jigar-arc10 - thank you for the additional testing.

Thoughts on some of the points raised above:

• These scripts required root access to the server.

Current Akash Provider documentation and install process assumes install is being run as root as stated here:

https://akash.network/docs/providers/build-a-cloud-provider/kubernetes-cluster-for-akash-providers/kubernetes-cluster-for-akash-providers/#step-2---install-ansible

As this is part of pre-existing methodologies - do not view this as an issue - but please let us know if you feel otherwise and/or if it will provoke issues in Praetor use.

• The recommended OS is Ubuntu. It failed for Debian at the ingress-nginx installation stage.

Current Akash Provider > Helm install based instructions recommend/assume Ubuntu use as stated here:

https://akash.network/docs/providers/build-a-cloud-provider/kubernetes-cluster-for-akash-providers/kubernetes-cluster-for-akash-providers/#kubernetes-cluster-softwarehardware-requirements-and-recommendations

Based on this being part of the pre-existing standard - do not believe this is an issue but please let us know if you feel otherwise and/or if this may cause issues for Praetor users.

• While scaling down a node, we tried to use the draining method, but akash-services/operator-inventory-hardware-discovery will cause an issue as it is not a DeamonSet. We should look into it. Force draining worked.

Will look into this issue further. Initial testing of scaling down procedure only tested the ability to scale down K3s nodes. Have not yet tested scaling down with Akash provider and related operators installed. Will test those scenarios ASAP.

[jigar-arc10]

@chainzero - Thanks for the response.

As this is part of pre-existing methodologies - do not view this as an issue - but please let us know if you feel otherwise and/or if it will provoke issues in Praetor use.

After deep consideration, we agree that root user access should be required as it also helps with GPU driver installation steps.

Based on this being part of the pre-existing standard - do not believe this is an issue but please let us know if you feel otherwise and/or if this may cause issues for Praetor users.

It's a non-issue.

Will look into this issue further. Initial testing of scaling down procedure only tested the ability to scale down K3s nodes. Have not yet tested scaling down with Akash provider and related operators installed. Will test those scenarios ASAP.

After many iterations of testing regarding node removal with updated scripts, the issue about operator-inventory-hardware is gone, and the node was successfully removed.