Closed ishioni closed 1 year ago
Thanks for reporting this. Could you elaborate on the numbers of PVCs and such that are involved here so I can try reproduce it? That 422 should probably be retried with a back-off before reporting the error to the CSI driver. The request above looks like it's an unmount that is being processed, not mount?
It's actually both, happens when i schedule the creation/deletion of many pods at the same time (many = 5), almost like the truenas middleware isn't catching up. Which is weird, since it's running on an i7-7700, with the nas almost in idle
I made a quick screengrab of a couple of things of interest whilst this is happening https://drive.google.com/file/d/1O_vrPwbf9LaCh1PXOMvYr_Hq4GzIU_O5/view?usp=sharing
Thanks for providing this. It goes without saying, control plane scale testing of any sort has not been performed. Which version of TrueNAS/FreeNAS is that?
22.02.04
Hey just to update, 22.12.0 fixed my kernel oops issue, but this still persists
Thanks for letting me know. I'm going to work on this over the Christmas break.
v2.3.0 just released. The CSP now handles more error conditions (422) more thoroughly.
I'm having a bit of stability problems on part of truenas (it seems scst is causing kernel panics). While debugging this, I've also noticed that truenas-csp likes to throw these around when trying to mount a bunch of pvcs at the same time. Any way to debug this further?