lease renewal failed error for GCP engine.

[duong-se]

Describe the bug

2024-05-23T03:09:03Z    DEBUG   events  Secret synced, lease_id="gcp/key/test_role/Q7Sgh0sRXfUq8dAAQhHNWqTS", horizon=17h24m34.836650247s, sync_reason="lease renewal failed"   {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"default","name":"test_service","uid":"be26dc7c-34f4-4ae4-8f75-7bbf20c2ec47","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1877679475"}, "reason": "SecretRotated"}

To Reproduce Steps to reproduce the behavior:

1. Config kubernetes auth method with role has ttl=24h and period=24h to get credentials from google cloud engine.
2. Deploy any service using google cloud credentials
3. See error (vault-secrets-operator logs)

Expected behavior lease renewal failed will not happen anymore

Environment

• Kubernetes version:
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): Google Cloud K8s version 1.27.12-gke.1115000
  • Other configuration options or runtime services (istio, etc.): No
• vault-secrets-operator version: 0.6.0

Additional context N/A

[duong-se]

@benashz I think not only the GCP renewal failed, the database dynamic secret has the same issue.