search

orange-cloudfoundry / k3s-wrapper-boshrelease

k3s wrapper scripts bosh release
Apache License 2.0
2 stars 2 forks source link

Add secret hosting, switch to `network.target`, kubelet root directory fix, introduce `k3s-utils`, and clean up ERB code #67

Closed kinjelom closed 3 weeks ago

kinjelom commented 1 month ago

Add Support for Hosting Secrets

This change allows you to store secrets in BOSH CredHub and place them in a hostPath to share with pods.

BOSH manifest example:

- type: replace
  path: /instance_groups/name=agent/jobs/name=k3s-agent/properties?/k3s?/host-secrets?/dir
  value: /var/vcap/data/secrets

- type: replace
  path: /instance_groups/name=agent/jobs/name=k3s-agent/properties?/k3s?/host-secrets?/files/-
  value:
    name: cloud.conf
    content: |+
      [Global]
      auth-url=((csi_cinder_auth_url))
      tls-insecure=((csi_cinder_tls_insecure))
      tenant-id=((csi_cinder_tenant_id))
      username=((csi_cinder_username))
      password=((csi_cinder_password))
      domain-name=((csi_cinder_domain))
      region=((csi_cinder_region))
      [BlockStorage]
      ignore-volume-az=((csi_cinder_ignore_volume_az))

K3s manifest example:

volumes:
- name: secret-cinderplugin
  hostPath:
    path: /var/vcap/data/secrets
    type: Directory

Switch to Using network.target in Systemd

An error in my deployment indicated that the systemd unit sys-subsystem-net-devices-eth0.device does not exist. In systemd, network devices are not always available as .device units. Instead, you can use a more general dependency like network.target, which ensures that network services are available. This change allows the service to start once the network is properly configured, without relying on a specific device unit that may not be present.

Kubelet root directory fix

Kubelet ignores the root-dir argument and creates the socket at: /var/lib/kubelet/device-plugins/=kubelet.sock Therefore, we linked /var/lib/kubelet to the specified root directory. https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-implementation

Other Changes

I introduced the k3s-utils package to share functions between the server and agent jobs (DRY) and cleaned up the ERB code.

kinjelom commented 1 month ago

When configuring both the K3s server and agent, it's crucial that they share the same --root-dir=/var/vcap/data/kubelet (properties) path because certain components, such as the CSI Cinder Node Plugin, run on both the agent and server. This requires consistent directory paths across the cluster. For example, the csi-cinder-nodeplugin DaemonSet needs to mount specific directories like /var/lib/kubelet/plugins/cinder.csi.openstack.org and /var/lib/kubelet/plugins_registry. If the server and agent are configured with different --root-dir paths (e.g., /var/vcap/data/k3s-server/kubelet vs /var/vcap/data/k3s-agent/kubelet), the plugin will encounter issues as it expects a single, unified path structure across the entire cluster.

o-orand commented 3 weeks ago

Thanks @kinjelom for your contribution. I'm sorry but we won't be able to review your PR until early September. I hope it will be okay for you.

poblin-orange commented 3 weeks ago

Hey @kinjelom . In addition to @o-orand points, it might help to split the PR to eases the review. probably erb and k3s-utils could be quickly reviewed. Distinct PR for:

Thks for your contrib !

kinjelom commented 3 weeks ago

@poblin-orange At the moment, I'm struggling with getting the CSI Cinder to work in this deployment (an issue with unmounting the volume when there are files on it – most likely a problem related to paths). Once I'm done, I'll return to the PR.

PS1. It seems to me that it would be best to merge both the k3s-server and k3s-agent jobs into a single k3s job that runs in either server or agent mode... this would solve the issue of redundant code (DRY) and the path inconsistency in kubelet – I’ll try to implement this and report back on whether this approach has potential. What about it?

PS2. Creating files using another boshrelease - it seems rather inconvenient to me. In fact, the ability to deliver such secrets should be one of the core functions of this boshrelease (considering that BOSH Credhub combined with a template seems like a good way to deliver secrets for system pods like the CSI nodeplugin).

poblin-orange commented 3 weeks ago

@kinjelom PS1: true that bosh does not help reusing files between jobs. I guess its by purpose ... We did choose two distinct jobs, as we rely heavily on bosh add ons to colocate other bosh release with server / or agent. Also we have bosh links from k3s-agent to k3s-server

PS2: putting the a file in a location seems very k3s agnostic. If what you want is populate a k8s Secret in the cluster, you can leverage https://github.com/orange-cloudfoundry/k3s-wrapper-boshrelease/blob/master/jobs/k3s-server/spec#L116 This lets you declaratively create any yaml in k3s, benefiting of credhub resolution

kinjelom commented 3 weeks ago

  1. Reusing files in BOSH is typically done through sources or packages. However, this is a separate topic, and I'll suggest something in a separate PR (if time permits...).

  2. Can I use additional-manifests for any text file, such as the one at https://github.com/kubernetes/cloud-provider-openstack/blob/master/pkg/csi/cinder/etc/cloud.conf, which can then be referenced in the CSI Cinder manifest? I can achieve this in another way, but the proposed solution seemed the most appropriate to me.

I successfully deployed Cinder CSI on this deployment, so I'm closing this PR and will break down the issues into separate PRs.

Thanks for the collaboration :)

kinjelom commented 3 weeks ago

@poblin-orange https://github.com/orange-cloudfoundry/k3s-wrapper-boshrelease/pull/69