Backup partially fails: error taking snapshot of volume: error reading from server: EOF

What steps did you take and what happened: I followed the various steps from velero and this repo documentations to create a remote snapshot location to a minio instance running on a machine outside the cluster.

I deployed a couple of services and tried to do a backup, unfortunately I am facing an error. The backup reports as Partially Failed and the logs shows that certain volumes couldn't be backed up with the following error:

time="2022-10-13T12:42:00Z" level=warning msg="Epoll wait failed : interrupted system call" backup=velero/test3 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/server.go
:302" pluginName=velero-blockstore-openebs

<... can be 50+ similar log entries and then ...>

time="2022-10-13T12:42:00Z" level=info msg="1 errors encountered backup up item" backup=velero/test3 logSource="pkg/backup/backup.go:413" name=registry-server-7d5466494d-84fj5
time="2022-10-13T12:42:00Z" level=error msg="Error backing up item" backup=velero/test3 error="error taking snapshot of volume: rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/backup/backup.go:4
17" name=registry-server-7d5466494d-84fj5

What did you expect to happen: I expect the backup to be successful

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)

kubectl logs deployment/velero -n velero -> here
kubectl logs deployment/maya-apiserver -n openebs -> N/A

velero backup describe <backupname>

Name:         test3
Namespace:    velero
Labels:       velero.io/storage-location=local-backup
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.2
            velero.io/source-cluster-k8s-major-version=1
            velero.io/source-cluster-k8s-minor-version=25

Phase:  PartiallyFailed (run `velero backup logs test3` for more information)

Errors:    4
Warnings:  165

Namespaces:
Included:  *
Excluded:  <none>

Resources:
Included:        *
Excluded:        <none>
Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  local-backup

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2022-10-13 13:41:29 +0100 BST
Completed:  2022-10-13 13:42:38 +0100 BST

Expiration:  2022-11-12 12:41:28 +0000 GMT

Total items to be backed up:  767
Items backed up:              767

Velero-Native Snapshots:  1 of 5 snapshots completed successfully (specify --details for more information)

CSI Volume Snapshots: <none included>

velero backup logs <backupname> -> here

Anything else you would like to add: Currently migrating a 25 nodes cluster to k8s. This is the initial setup/test before definitive migration with just 5 nodes, self-hosted microk8s cluster running cStor

Environment:

Velero version (use velero version):

Client:
      Version: v1.9.2
      Git commit: -
Server:
      Version: v1.9.2

Velero features (use velero client config get features): features: EnableCSI
Velero-plugin version: 3.3.0
OpenEBS version: 3.3.0
Kubernetes version (use kubectl version): 1.25
Kubernetes installer & version:
Cloud provider or hardware configuration: 5 nodes, home made from consumer products, bought over the years so nodes are slightly different
- Intel i7 or i9 from 8th to 10th Gen
- 16GB to 32GB
- 2 nodes have 3x 1TB SSD for cStor

OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

openebs / velero-plugin

Backup partially fails: error taking snapshot of volume: error reading from server: EOF #182