Sometimes GcpNfsVolumeRestore get applied twice

[abalaie]

Description

Sometimes, the restore happens twice which puts the filestore in an unusable state twice the actual time it needs to be. This can be few extra minutes of unnecessary operation and cost.

Expected result

Actual result

Steps to reproduce

Troubleshooting

[abalaie]

This issue happens because restore reconciler is interrupted after running restore and before saving the operationId on gcpnfsvolumerestore object status.

This is proposed way of fixing it:

• Create a lease/lock mechanism so only one restore can be active per gcpNfsVolume at a time.
• Introduce a find restore method to search running "restore" operations for a given gcpNfsVolume

Replace these steps in gcpNfsVolumeRestore reconciler:

• If operation id is empty, run the restore and set the operation id
• Check the operation id. If not complete requeueWithDelay

with these steps:

• Lease the destination gcpNfsVolume. If already leased by itself, renew the lease (update the renew timestamp). If unsuccessful (another restore has the lease) try again after delay.
• search for any running restore operation for the destination gcpNfsVolume. If found, assume the ownership and add the operation id to status
• If operation id is empty, run the restore and set the operation id
• Check the operation id. If not complete requeueWithDelay
• Release the lease

POC already done and it worked.