Open HanssenKai opened 1 year ago
Hey @tuxtof, we talked briefly in kubecon amsterdam about this issue, but i went with mayastor to quickly get something working. Got some time to look at it again and finally got it working with talos.
To give some more context around talos, here is a issue to get longhorn working on talos and documentation. The problem is due to talos having very few binaries and is immutable without package manager or shell capabilities. While it does make for a very smooth kubernetes experience, in particular CSI implementations tend to encounter issues because they make assumptions about host node capabilities.
Hope this can be a starting point to get the general idea if you ever want to support talos in the future, or if other lost souls ever find themselves on the same path
The rough implentation to get it to work was;
nutanix-csi-node
with missing utilities /chroot-host-wrapper.sh
in nutanix-csi-node
to run most commands through the utilities sidecar utiliteis container with nsenter
, remembering to redirect paths like /dev
and /var
to the host node /dev
and /var
e.g mkfs.xfs /host/dev/disk/by-id...
/usr/sbin/[mount/umount/xfs_growfs]
with wrapper script to run the commands on kubelet
with nsenter
. Otherwise pods won't have the volume mounted correctly. /usr/sbin/[iscsiadm/iscsi]
with wrapper script to run through the iscsi extension with nsenter
Utilities sidecar:
FROM alpine
# Update apk and install necessary packages
RUN apk update
RUN apk add --no-cache \
procps \
xfsprogs \
xfsprogs-extra \
util-linux \
nfs-utils \
multipath-tools \
procps \
&& rm -rf /var/cache/apk/*
# Use host as workdir
WORKDIR /host
# Set a default command to run an infinite sleep loop
CMD while true; do \
echo $(pgrep iscsid) > /etc/shared-data/iscsid; \
echo $(pgrep kubelet) > /etc/shared-data/kubelet; \
echo $$ > /etc/shared-data/utils; \
sleep 30; \
done;
wrapper scripts configmap:
data:
iscsiadm: |
#!/bin/sh
iscsid_pid=$(cat /etc/shared-data/iscsid)
nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@"
iscsi: |
#!/bin/sh
iscsid_pid=$(cat /etc/shared-data/iscsid)
nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsi "$@"
mount: |
#!/usr/bin/env bash
ME=`basename "$0"`
kubelet_pid=$(cat /etc/shared-data/kubelet)
nsenter --target=${kubelet_pid} --mount -- "${ME}" "${@:1}"
utils: |
#!/usr/bin/env bash
ME=`basename "$0"`
utils_pid=$(cat /etc/shared-data/utils)
#nsenter --mount="/proc/${utils_pid}/ns/mnt" --net="/proc/${utils_pid}/ns/net" -- "${ME}" "${@:1}"
# Transform all occurrences of '/dev' in arguments to '/host/dev'
# This processes all arguments, replacing each instance as needed
args=("$@")
for i in "${!args[@]}"; do
args[$i]=$(echo "${args[$i]}" | sed 's|/dev|/host/dev|g')
args[$i]=$(echo "${args[$i]}" | sed 's|/var|/host/var|g')
done
# Use nsenter to run the command in the target namespaces with the transformed arguments
nsenter --target=${utils_pid} --mount -- "${ME}" "${args[@]}"
Hitting a roadblock trying to implement nutanx csi on a talos cluster, where the nutanix csi node is unable to run either
mkfs.ext4
ormkfs.xfs
as they are not provided by talos. Would it be possible to bundle some of these tools in the container image?