velero-restore-helper hangs forever when rebooting cluster twice

[RobKenis]

What steps did you take and what happened:

• Create a Velero backup using filesystem backup, backed by Kopia
• Restore the Velero backup on a different server. This restore is successful.
• Reboot the new server
• After the reboot, validate that all Pods go into the Unknown state. After that, all Containers are restarted and Pods are again in the Running state.
• Reboot the new server again
• After reboot, some containers remain in the Init: 0/1 state with following logs. The .velero directory is no longer present in the top-level of the volume, so it makes sense that the file will never be found.

The filesystem restore done file /restores/repo1/.velero/5132d6c3-7290-45a2-a04a-f4f851a2642e is not found yet. Retry later.
The filesystem restore done file /restores/repo1/.velero/5132d6c3-7290-45a2-a04a-f4f851a2642e is not found yet. Retry later.

What did you expect to happen:

Either the .velero directory is not cleaned up, this change was made in this change, or the restore-helper can handle multiple reboots.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

bundle-2024-07-16-15-57-22.tar.gz

Anything else you would like to add:

Environment:

• Velero version (use velero version):

  Client:
      Version: v1.14.0
      Git commit: 2fc6300f2239f250b40b0488c35feae59520f2d3
  Server:
      Version: v1.13.2

• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version):

  Client Version: v1.28.2+k3s1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.2+k3s1

• Kubernetes installer & version: k3s
• Cloud provider or hardware configuration: VM on libvirt
• OS (e.g. from /etc/os-release):

VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[reasonerjt]

@RobKenis Let me clarify when you say Reboot the new server you mean rebooting the api-server right? Is this an expected behavior?

[kaovilai]

Would it be possible for restic restore helper to detect recent reboot during init?

[RobKenis]

@reasonerjt By rebooting the server, I mean running reboot. So the entire OS restarts, including the api-server.