Open brunnels opened 3 months ago
Hi @brunnels,
Can you exec the command "dd if=/dev/sdb bs=4096 count=512 status=none" on the node ?
The function just execute the above command and get the result.
The node is talos so no ssh access. I can exec on the trident-node-linux pod if that would help but logs indicate it would probably fail. All my talos nodes have the iscsi-tools system extension.
@davidcheng0716 It looks like the synology-csi that uses iscsi for RWO and cifs for RWM works well under talos. https://www.talos.dev/v1.7/kubernetes-guides/configuration/synology-csi/
Perhaps the qnap-csi needs some additional configuration steps like the openebs jiva as detailed here?
@davidcheng0716 I found this project talos-debug-tools that creates a daemonset similar to the qnap csi driver. I changed the namespace to be the same as the qnap csi driver namespace and I am able to ssh into the pod and run the dd command without error.
Maybe the debug daemonset could provide some insight into something the qnap csi daemonset might be missing?
Comparing the daemonset-trident-node-linux
and the debug daemonset the only things that pop out is the pod securityContext
is this in the debug daemonset
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
and this for the spec.template.spec.securityContext
securityContext:
runAsNonRoot: false
seccompProfile:
type: RuntimeDefault
Hi @brunnels, We have received your report and suspect the issue may be caused by a specific Linux distribution. We are actively working to resolve this problem.
@davidcheng0716 Do you plan to release the source for the operator? If the source were available I could make the necessary changes and test.
@davidcheng0716 any movement on this? I'm happy to assist with testing.
@brunnels Sorry for the late reply, and I'll take a look at this issue. To reproduce the issue, I'm going to build the same environment; what kind of installed talos do you have? Bare metal, virtualbox, or others?
@davidcheng0716 I have baremetal. I'm using https://github.com/onedr0p/cluster-template template. This makes it very quick to get a repeatable, secure, and best practices cluster up in a few minutes.
@brunnels I guess the problem is the node doesn't have the tool 'dd'; our service doesn't have 'dd' in the container.
The talos debug daemonset installs the tools in the container instead of calling the tool on the node. Therefore, it can work when you ssh into the pod.
I'm wondering if the node has the tool 'dd' without the container on the node.
@davidcheng0716 no, that's not the problem. If you see my prior comment I can use the talos debug daemonset to get into a pod in the same namespace on the same node and I can run dd on the iscsi pvc that qnap created. I think the issue lies with the pod securityContext.
By default talos security policy prevents privileged escalation in the pod but can be overridden with a namespace annotation and adding the capabilites https://www.talos.dev/v1.7/kubernetes-guides/configuration/pod-security/
I have the annotation on the qnap-trident namespace but I cannot change the pod securityContext set but the qnap-trident operator.
@davidcheng0716 I just re-read you comment and looked at the debug-tools dockerfile and you are correct. It is installing dd into the pod. My apologies.
The node won't have dd. It's the very bare minimum so anything the pod needs it must provide.
Latest QTS5 version
Logs from trident controller don't appear to have anything wrong
trident-node-linux logs have errors