search

vmware-tanzu / astrolabe

Data protection framework for complex applications
Other
38 stars 21 forks source link

Disconnected vCenter reconciliation #73

Closed deepakkinni closed 3 years ago

deepakkinni commented 3 years ago

During the data-manager and backupdriver startup, an attempt is made to initialize ivd petm. Before the fix, the ivd petm would not be registered if there were errors while establishing a vCenter connection. Even if the vc comes back online, there was no mechanism to re-register ivd petm. With this fix, the ivd petm is registered even if there were errors while initializing ivd petm. Furthermore, the vcenter connections are checked before any vCenter operation, and connections are re-established if no present.

Confluence: https://confluence.eng.vmware.com/display/~dkinni/Disconnected+VC+Connection+Reconciliation+Velero+vSphere+Plugin

Testing Please refer https://confluence.eng.vmware.com/display/~dkinni/Disconnected+VC+Connection+Reconciliation+Velero+vSphere+Plugin for all the testing done.

Signed-off-by: Deepak Kinni dkinni@vmware.com

deepakkinni commented 3 years ago

ReloadConfig for each API call could be too heavy and introduce significant performance impact in green path. Can we only retry on the failure of API call due to loss of connection?

It doesn't add any significant performance hit, it makes a bunch of nil checks and some string comparisons, there are absolutely no network calls made.

Also, IIRC, in snapshot manager of plugin side, the ivdPETM reference will be nil if VC connection is not initialized properly. Can we only trigger ReloadConfig if the ivdPETM reference is nil? ivdPETM will never be nil based off this change, there will be an ivdPETM with the vCenter as nil though.

dsu-igeek commented 3 years ago

Also, there's at least on use of vslmManager from inside ivd_protected_entity_manager that (GetProtectedEntities) that should be checking/fixing the connection state.

deepakkinni commented 3 years ago

How does the password rotation get detected/handled?

The password rotation is handled from the plugin, the plugin watches a Secret and pushes the new credentials into astrolabe.